WebShop

Posted on In Views:
About Webshop Benchmark

1. Paper Reading

What is WebShop?

WebShop is a large-scale web-based ==interactive environment== with over 1.1 million real-world products scraped from amazon.com.

1.1 Task

WebShop can be formulated as a partially observable Markov decision process.

  • State Space: 𝒮
  • Action Space: 𝒜
  • Deterministic Decision Function: 𝒯 : 𝒮 × 𝒜 → 𝒮
  • Reward Function: ℛ : 𝒮 × 𝒜 → [0, 1]
  • Instruction Space: 𝒰
  • State Observation Space: 𝒪

1.1.1 State & Action

A state 𝓈 ∈ 𝒮 represents a web page, which falls into one of the four types:

  • 𝓈ℯ𝒶𝓇𝒸𝒽 page: contains a research bar.
  • 𝓇ℯ𝓈𝓊𝓁𝓉𝓈 page: lists a set of products returned by research engine.
  • 𝒾𝓉ℯ𝓂 page: describes a product.
  • 𝒾𝓉ℯ𝓂 − 𝒹ℯ𝓉𝒶𝒾𝓁 page: shows further information about the product.

For a product 𝓎, author denotes:

  • 𝓎̂: aggregation of the various text fields including product title, description, and overview
  • 𝓎𝓅𝓇𝒾𝒸ℯ: price
  • Yopt: set of buying options
  • I: images, each corresponding to a specific option
  • Yatt: set of attributes hidden from the agent which is extracted from the title and the item − detail pages

Available actions 𝒶 ∈ 𝒜(𝓈) :

Generally, there are two action types: search and choose. When agent is at search page, the only action allowed is search. On other pages, click is the only action choice.

The chosen action argument (button) will be clicked as a web link as opposed to the low-level mouse-click actions in previous environments such as World of Bits.

The transitions initiated by clicks deterministically redirect the web page to one of the four page types (Table 1). The transition initiated by search is based on a deterministic search engine.

1.1.2 Observation

Using Flask and OpenAI Gym, author provides two parallel observation modes to render the state and instruction 𝒮 × ℐ = 𝒪.

  1. HTML Mode: contains the HTML of the web page, allowing for interaction in a web browser.
  2. Simple Mode: strips away extraneous meta-data from raw HTML into a simpler format.

Human Performance Scores are collected in HTML mode, while all models are trained and trained and evaluated in the simple mode.

==Note==:

​ Though environment allows for training reinforcement learning agents on raw pixels in HTML mode. Author believes that it provides a very low-level non-semantic action space.

​ Moreover, it is straightforward to write a translator that converts any new HTML page into simple format for use with trained agents, which enables sim-to-real transfer.

1.1.3 Instruction & Reward

Each natural language instruction 𝓊 ∈ 𝒰 contains the following information: a non-empty set of attributes Uatt, a set of options Uopt, and a price uprice. The instruction is generated based on a target product ŷ by human annotators.

Concretely,

  • Uatt ⊆ Yatt* is a subset of the product attributes
  • Uopt ⊆ Yopt* is a subset of the product option field-value pairs
  • uprice > yprice* is a price set to be higher than the target product price

e.g.

Instruction:

“Can you find me a pair of black-and-blue sneaker that is good in rain weather? I want it to have puffy soles, and price less than 90 dollars.”

The instruction contains the aforementioned attributes and options:

  • Uatt = “waterproof”, “soft sole”
  • Uopt = {“color”: “black and blue”}

In each episode, the agent receives a reward 𝓇 = ℛ(𝓈𝒯, 𝒶) in the end at timestep T, where a = choose[buy], y is the product chosen by the agent in the final state sT, and Yatt and Yopt are its corresponding attributes and options. The reward is defined as:

$$ r = r_{\mathrm{type}} \cdot \frac{ |U_{\mathrm{att}} \cap Y_{\mathrm{att}}| + |U_{\mathrm{opt}} \cap Y_{\mathrm{opt}}| + \mathbf{1}[y_{\mathrm{price}} \leq u_{\mathrm{price}}] }{ |U_{\mathrm{att}}| + |U_{\mathrm{opt}}| + 1 } $$

In fact the reward is designed to calculate the ==fraction of constraints satisfied==.

$$ r = r_\mathrm{type} \cdot \frac {\mathrm{satisfied\ attributes}+\mathrm{satisfied\ options}+\mathbf{1}[\mathrm{price\ satisfied}]} {\mathrm{attributes + options + 1}} $$

The type reward $r_\mathrm{type} = \text{TextMatch}(\overline y, \overline y^*)$ is based on text matching heuristics to assign low reward when y and y* have similar attributes and options but are obviously different types of products. For example, “butter” and “plant-based meat” differ in types but may both contain attributes “cruelty-free”, “non-GMO”, and an option “size: pack of 2”.

1.1.4 Evaluation Metrics

There are 2 evaluation metrics:

    1. Task Score defined as 100 × avg.reward, which captures the average reward obtained across episodes;
    1. Success Rate defined as the portion of instructions where r = 1. ==Note== that it is possible to obtain r = 1 for an episode even if the final product is not y* — there could be many items that satisfy the goal “I want a red shirt”, even if the goal is generated from a specific red shirt item.

1.2 Environment

1.2.1 Data Scraping

Author uses ==ScraperAPI== to scrape 1, 181, 436 products from amazon.com across 5 categories (fashion, makeup, electronics, furniture, and food) using 113 sub-category names as queries. The product texts (title and item details) have an average length of 262.9 and a vocabulary size 224, 041 (word frequency higher than 10). In addition, the products have a total of 842, 849 unique options, reflecting the scale and complexity of the data. More details about product scraping is in the Appendix §A.1.

1.2.2 Search Engine

Author uses ==Pyserini== for the search engine, where indices are built offline using a ==BM25== sparse retriever with text for each product concatenated from the title, description, overview, and customization options. The search engine is deterministic, which eases imitation learning and result reproducibility. More details in A.3.

1.2.3 Attribute Mining and Annotation

Each product is annotated with a set of hidden attributes, which are used to represent its latent characteristics as well as to calculate the reward as detailed in §3. An attribute is a short natural language phrase that describes the property of the product (see examples in Figure 1). Author mines the attributes by calculating ==TF-IDF== scores for all bi-grams in the concatenated titles and descriptions based on each product category. Author reviews the top 200 bi-grams for each category, remove the noisy ones by inspection (decide based on whether the bi-gram is human understandable), and assign them to the products. Author consolidates a pool of 670 attributes. See more details in the Appendix §A.2.

1.2.4 Natural Language Instructions

Author uses ==Amazon Mechanical Turk== (AMT) to collect natural language instructions that specify goal products with appropriate options. Specifically, an AMT worker is presented with a sampled goal product, including the product title, category, attributes, and the buying options, and asked to write a command to instruct an automatic shopping agent to find the target. Workers are instructed to ==avoid== being too specific such as including the entire title in the instruction, but stay faithful to describing the target product. Author collects a total of 12, 087 linguistically diverse instructions with an overall vocabulary size of 9, 036 words and an average length of 15.9 words. Author provides the detailed annotation process and interface in the Appendix §A.4.

1.2.5 Human Demonstrations

Author collects trajectories from humans performing the task in the HTML mode of WebShop to understand the task difficulty for humans and to analyze how humans would solve the task. Author uses qualification tests to train and select motivated workers to perform the task. Author recruits and train a total of 13 workers for data collection, and among them author selects the top 7 performing workers to be “experts” (see Appendix §A.6 for examples). Author also leverages this data to perform imitation learning (described in §4.2).

1.3 Training

Omit.

1.4 Experiments

Omit.

2. Engineering

2.1 Workflow

2.1.1 Basic Environment Preparation

1
2
3
4
5
conda create -n webshop python=3.8.13
conda install -c conda-forge openjdk=11

pip install -r requirements.txt
python -m spacy download en_core_web_sm

2.1.2 Data Preparation

2.1.2.1 setup.sh

setup.shmainly downloads Item info, Instructions, NLP model and builds Item Index.

1
./setup.sh -d small # or -d all

small means 1000 item information; all means all items.

setup.sh:

1
2
3
4
5
6
7
8
9
10
11
12
13
mkdir -p data
cd data
if [ "$data" == "small" ]; then
	# items_shuffle_1000.json; Info including title description price option... of items
	gdown https://drive.google.com/uc?id=1EgHdxQ_YxqIQlvvq5iKlCrkEKR6-j0Ib;
	# items_ins_v2_1000.json; attributes + instruction
	gdown https://drive.google.com/uc?id=1IduG0xl544V_A_jv3tHXC0kyFi7PnyBu;
	
elif [ "$data" == "all" ]; then
	# items_shuffle.json
	gdown https://drive.google.com/uc?id=1A2whVgOO0euk5O13n2iYDM0bQRkkRduB;\
	# items_ins_v2.json
	gdown https://drive.google.com/uc?id=1s2j6NgHljiZzQNL3veZaAiyW_qDEgBNi;
1
2
# items_human_ins.json; human shopping instructions
gdown https://drive.google.com/uc?id=14Kb5SPBk_jfdLZ_CDBNitW98QLDlKR5O

Download a NLP model; When calculating rewards, it is needed:

1
2
# Download spaCy large NLP model
python -m spacy download en_core_web_lg

Build research engine index:

When searching items, it is unrealistic to retrieve all items which costs a lot of time. Lucene builds a index:

[word -> lists of items including the word].

1
2
3
4
5
6
7
8
# Build search engine index
cd search_engine
# resources_100 for 100 items, resources_1k for 1k items
# resources_100k for 100k items, 
mkdir -p resources resources_100 resources_1k resources_100k
python convert_product_file_format.py # convert items.json => required doc format
mkdir -p indexes
./run_indexing.sh

python convert_product_file_format.py: read data/items_shuffle_1000.json and convert each product into a single plain-text string optimized for search.

convert_product_file_format.py:21-28:

1
2
3
4
5
6
7
8
9
10
doc = dict()
doc['id'] = p['asin']
doc['contents'] = ' '.join([
    p['Title'],
    p['Description'],
    p['BulletPoints'][0],
    option_text,
]).lower()
doc['product'] = p
docs.append(doc)

converted item example:

1
2
3
4
5
{
    "id": "B0024...",
    "contents": "mens casual sneakers ... color: black, white | size: 9, 10 ,11 ..."
    "product": {...} 
}

2.1.3 Environment

2.1.3.1 Main Env

web_agent_site/envs/web_agent_text_env.py:

  1. Standard Library

1
2
3
4
5
6
7
8
import gym
import json
import random
import string
import time
import torch

import numpy as np

  • gym: to rea

2.1.3.2 Engine

web_agent_site/engine/engine.py

2.1.3.3 Task & Reward

web_agent_site/engine/goal.py

2.1.3.4 Others

web_agent_site/utils.py

本文链接：https://wangyier.top/webshop/

版权声明：本博客所有文章除特别声明外，均采用 CC BY-NC-SA 4.0 许可协议。转载请注明来自 The Great Library！