Note:Scenarios created from FileStore objects cannot be used with question memory rules, and can only be added to questions with the by() method, not the loop() method. This is because the memory rules and loop() method insert the filepath in the question, whereas the by() method inserts the file content when the question is run. See details on these methods at the Scenarios section of the documentation.
File types
The following file types are currently supported by the FileStore module:- docx (Word document)
- csv (Comma-separated values)
- html (HyperText Markup Language)
- json (JavaScript Object Notation)
- latex (LaTeX)
- md (Markdown)
- pdf (Portable Document Format)
- png (image)
- pptx (PowerPoint)
- py (Python)
- sql (SQL database)
- sqlite (SQLite database)
- txt (text)
Posting a file
- Import the FileStore constructor and create an object by passing the path to the file. The constructor will automatically infer the file type from the extension. For example:
- Call the push method to post the file at Coop. You can optionally pass the following parameters:
- description: a string description for the file
- alias: a convenient Pythonic reference name for the URL for the object, e.g., my_example_csv
- visibility: either public, private or unlisted (the default is unlisted)
- description: the description you provided, if any
- object_type: the type of object (e.g., scenario, survey, results, agent, notebook; objects posted with FileStore are always scenarios)
- url: the URL of the file at Coop
- uuid: the UUID of the file at Coop
- version: the version of the file
- visibility: the visibility of the file (public, private or unlisted by default)
Retrieving a file
To retrieve a file, call the pull method on the FileStore constructor and pass it the alias or UUID of the file that you want to retrieve. For the example above, we can retrieve the file with:CSV example
Here we create an example CSV and then post it to Coop using FileStore and retrieve it. Then we use the retrieved file to construct scenarios for questions (you can skip the step to create a CSV and replace with your own file). To create an example CSV file:Age | City | Occupation |
---|---|---|
25 | New York | Software Engineer |
30 | San Francisco | Teacher |
35 | Chicago | Doctor |
28 | Boston | Data Scientist |
45 | Seattle | Architect |
PNG example
Here we post and retrieve an image file, and then create a scenario for it. Note that we need to specify the scenario key for the file when we create the scenario. We also need to ensure that we have specified a vision model when using it with a survey (e.g., gpt-4o). To post the file:model | scenario.logo | answer.test |
---|---|---|
gpt-4 | FileStore: self.path | The logo features a large, stylized letter “E” in a serif font on the left. Next to it, within square brackets, is a colorful parrot. The parrot has a green body, an orange beak, a pink chest, blue lower body, and gray feet. The design combines a classic typographic element with a vibrant, playful illustration. |
PDF example
Here we download an example PDF from the internet, post and retrieve it from Coop using FileStore and then convert it into a ScenarioList object with the from_pdf() method. The default keys are filename, page, text, which can be modified with the rename method. To download a PDF file:FileStore class
Bases:class edsl.scenarios.FileStore(path: str | None = None, mime_type: str | None = None, binary: bool | None = None, suffix: str | None = None, base64_string: str | None = None, external_locations: Dict[str, str] | None = None, extracted_text: str | None = None, **kwargs)[source]
Scenario
A specialized Scenario subclass for managing file content and metadata.
FileStore provides functionality for working with files in EDSL, handling various file formats with appropriate encoding, storage, and access methods. It extends Scenario to allow files to be included in surveys, questions, and other EDSL components.
FileStore supports multiple file formats including text, PDF, Word documents, images, and more. It can load files from local paths or URLs, and provides methods for accessing file content, extracting text, and managing file operations.
Key features: - Base64 encoding for portability and serialization - Lazy loading through temporary files when needed - Automatic MIME type detection - Text extraction from various file formats - Format-specific operations through specialized handlers
Attributes:_path (str): The original file path. _temp_path (str): Path to any generated temporary file. suffix (str): File extension. binary (bool): Whether the file is binary. mime_type (str): The file’s MIME type. base64_string (str): Base64-encoded file content. external_locations (dict): Dictionary of external locations. extracted_text (str): Text extracted from the file.
Examples:
Initialize a new FileStore object. This constructor creates a FileStore object from either a file path or a base64-encoded string representation of file content. It handles automatic detection of file properties like MIME type, extracts text content when possible, and manages file encoding.init(path: str | None = None, mime_type: str | None = None, binary: bool | None = None, suffix: str | None = None, base64_string: str | None = None, external_locations: Dict[str, str] | None = None, extracted_text: str | None = None, **kwargs)[source]
Args:path: Path to the file to load. Can be a local file path or URL. mime_type: MIME type of the file. If not provided, will be auto-detected. binary: Whether the file is binary. Defaults to False. suffix: File extension. If not provided, will be extracted from the path. base64_string: Base64-encoded file content. If provided, the file content will be loaded from this string instead of the path. external_locations: Dictionary mapping location names to URLs or paths where the file can also be accessed. extracted_text: Pre-extracted text content from the file. If not provided, text will be extracted automatically if possible. **kwargs: Additional keyword arguments. ‘filename’ can be used as an alternative to ‘path’.
This method uploads a file to Google’s Generative AI service asynchronously, polls for activation status with exponential backoff, and returns the file info.
Args:
refresh: If True, force re-upload even if already uploaded
Returns:
Dictionary containing the Google file information
Raises:
Exception: If upload fails or file activation fails
static base64_to_file(base64_string, is_binary=True)[source]
static base64_to_text_file(base64_string) → IO[source]
Take screenshots of multiple URLs concurrently. Args: urls: List of URLs to screenshot **kwargs: Additional arguments passed to screenshot function (full_page, wait_until, etc.) Returns: ScenarioList containing FileStore objects with their corresponding URLsclassmethod batch_screenshots(urls: List[str], **kwargs) → ScenarioList[source]
create_link(custom_filename=None, style=None)[source]
download_link()[source]
encode_file_to_base64_string(file_path: str)[source]
Returns an example Scenario instance.classmethod example(example_type=‘txt’)[source]
Args:randomize: If True, adds a random string to the value of the example key to ensure uniqueness.
Returns:A Scenario instance with example data suitable for testing or demonstration.
Examples:
extract_text() → str[source]
Creates a Scenario from a dictionary, with special handling for FileStore objects. This method creates a Scenario using the provided dictionary. It has special handling for dictionary values that represent serialized FileStore objects, which it will deserialize back into proper FileStore instances.classmethod from_dict(d)[source]
Args:d: A dictionary to convert to a Scenario.
Returns:A new Scenario containing the provided dictionary data.
Examples:
Notes:
- Any dictionary values that match the FileStore format will be converted to FileStore objects
- The method detects FileStore objects by looking for “base64_string” and “path” keys
- EDSL version information is automatically removed by the @remove_edsl_version decorator
- This method is commonly used when deserializing scenarios from JSON or other formats
classmethod from_url(url: str, download_path: str | None = None, mime_type: str | None = None) → FileStore[source]
Parameters:
- url – The URL of the file to download.
- download_path – The path to save the downloaded file.
- mime_type – The MIME type of the file. If None, it will be guessed from the file extension.
Synchronous wrapper for screenshot functionalityclassmethod from_url_screenshot(url: str, **kwargs) → FileStore[source]
Get the dimensions (width, height) of an image file.get_image_dimensions() → tuple[source]
Returns:tuple: A tuple containing the width and height of the image.
Raises:ValueError: If the file is not an image or PIL is not installed.
Examples:
Get metadata about a video file such as duration, dimensions, codec, etc. Uses FFmpeg to extract the information if available.get_video_metadata() → dict[source]
Returns:dict: A dictionary containing video metadata, or a dictionary with error information if metadata extraction fails.
Raises:ValueError: If the file is not a video.
Examples:
Check if the file is an image by examining its MIME type.is_image() → bool[source]
Returns:bool: True if the file is an image, False otherwise.
Examples:
Check if the file is a video by examining its MIME type.is_video() → bool[source]
Returns:bool: True if the file is a video, False otherwise.
Examples:
Offloads base64-encoded content from the FileStore by replacing ‘base64_string’ with ‘offloaded’. This reduces memory usage.
Args:inplace (bool): If True, modify the current FileStore. If False, return a new one.
Returns:FileStore: The modified FileStore (either self or a new instance).
open() → IO[source]
Returns a valid path to the file content, creating a temporary file if needed. This property ensures that a valid file path is always available for the file content, even if the original file is no longer accessible or if the FileStore was created from a base64 string without a path. If the original path doesn’t exist, it automatically generates a temporary file from the base64 content.property path*: str*[source]
Returns:A string containing a valid file path to access the file content.
Examples:
Notes:
- The path may point to a temporary file that will be cleaned up when the Python process exits
- Accessing this property may create a new temporary file if needed
- This property provides a consistent interface regardless of how the FileStore was created (from file or from base64 string)
Pull a FileStore object from Coop. Args: url_or_uuid: Either a UUID string or a URL pointing to the object expected_parrot_url: Optional URL for the Parrot server Returns: FileStore: The pulled FileStore objectclassmethod pull(url_or_uuid: str | UUID) → FileStore[source]
Push the object to Coop. :param description: The description of the object to push. :param visibility: The visibility of the object to push.push(description: str | None = None, alias: str | None = None, visibility: str | None = ‘unlisted’, expected_parrot_url: str | None = None) → dict[source]
Saves the FileStore’s file content to a Google Cloud Storage bucket using a signed URL. Args: signed_url (str): The signed URL for uploading to GCS bucket Returns: dict: Response from the GCS upload operation Raises: ValueError: If base64_string is offloaded or missing requests.RequestException: If the upload failssave_to_gcs_bucket(signed_url: str) → dict[source]
property size*: int*[source]
property text[source]
Convert the file content to a pandas DataFrame if supported by the file handler. Returns: pandas.DataFrame: The data from the file as a DataFrame Raises: AttributeError: If the file type’s handler doesn’t support pandas conversionto_pandas()[source]
to_scenario(key_name: str | None = None)[source]
to_tempfile(suffix=None)[source]
upload_google(refresh: bool = False) → None[source]
Display an interactive visualization of this object. Returns: The result of the dataset’s view methodview() → None[source]
Write the file content to disk, either to a specified filename or a temporary file. Args: filename (Optional[str]): The destination filename. If None, creates a temporary file. Returns: str: The path to the written file.write(filename: str | None = None) → str[source]