2DO Aggregator

The PHP port of python 2do-server.

Before download or install : the easiest way to use this is to get the teleport board in-world (hop://speculoos.world:8002:Lab) and ask us to include your calendar in 2do.directory.

Aggregator is a tool to fetch events from various sources and export them on several format, for use by 2do Board and other applications related with 2do Events project.

Import formats:

iCal (ics, iCalendar)
Custom web parsers:
- opensimworld
- thirdrock
- gridtalk (to do)
- OpenSim Fest (to do)

Export formats:

HYPEvent (legacy 2do/HYPEvent format)
- events.lsl2 (for current versions of 2do-board)
- events.lsl (old format, now includes only a deprecation notice)
JSON (events.json) compatible with events parsers (provided by w4os or Flexible Helper Scripts)
iCal (events.ics) iCalendar format, compatible with web, mobile and desktop calendars
Light html web calendar page for standalone use

This is a side PHP application intended to provide the same functionality as 2do-server, the original events fetcher of 2do project and the original HYPEvents code, but in a more modular and maintainable way, and with better integration with other tools of 2do Events, w4os and Flexiple Helper Scripts projects.

2do-board: the in-world board to display the calendar

For grid owners :

W4OS: a WordPress plugin to manage OpenSimulator grids and provide external helpers, including 2do Events
Flexible Helper Scripts: a standalione collection of scripts providing the same helpers, without the web interface

Getting started (recommended)

Don't install this app. Seriously. In most cases you don't need to install it, you can use ours.

as a parcel owner, you only need the the in-world board, which is intended to be used on any grid. It is available here:
- Speculoos grid, Lab region speculoos.world:8002:Lab
- or for scripters/builders: 2DO board Github repository,
as grid or region owner, you can use 2do.directory (https://2do.directory) service to enable events search on your grid with a simple straight-forward configuration
2do.directory service is also included by w4os WordPress Interface for OpenSimulator plugin

Jump directly to "Calendar conventions" below for the events format.

Installation

Installation of this server is only relevant if you want to provide a custom-curated list calendar. If you really need to manage your own calendars collection, follow these instructions.

Clone this repository and put it in a convenient place like /opt/2do-aggregator (not inside the website root folder).

Install libraries.

  cd /opt/2do-aggregator
  composer install --no-dev

Why outside the website directory? This is a script intended to be run from terminal or via a cron job, there is no point allowing random users or bots to run it from outside and risk overloading the server.

Copy config/ical.gfg.example and add your calendar sources.

  cp config/sources.csv.example config/sources.csv

Calendar conventions

Events must have

A title
A start and end date/time
A location composed of the region HG url (e.g. yourgrid.org:8002:My_Region)

They might also include

A description (optional but recommended)
A category (optional), following standards recognized by the viewers:
- discussion
- sports
- live music
- commercial
- nightlife/entertainment
- games/contests
- pageants
- education
- arts and culture
- charity/support groups
- miscellaneous
- These aliases are also recognized as variants of standard categories:
- art (art and culture)
- lecture (art and culture)
- litterature (art and culture)
- fair (nightlife/entertainment)
- music (life music)
- roleplay (games/contests)
- social (charity/support groups)

How to export Google Calendar as .ics

To get the url of your calendar in iCal format, move your mouse above the calendar you want to share, a three dots icon appears, select "Settings and Sharing" and scroll the page down to find Public iCal format adress. This is the value you need to copy as calendar ics url.

Running

Run the script manually

  /opt/2do-aggregator/aggregator.php /var/www/html/events/

Create a cronjob to run automatically (below example would run it every 4 hours)

  0 */4 * * * /opt/2do-aggregator/aggregator.php /var/www/html/events/

Assuming /var/www/html is your website root directory, and http://www.yourgrid.org/, this would create:

http://www.yourgrid.org/events/ a basic web calendar page
http://www.yourgrid.org/events/events.lsl2 the source url for 2do Board
http://www.yourgrid.org/events/events.json the source url for events parsers
- w4os Web interface for OpenSimulator (wordpress plugin + parsers)
- Flexible Helper Scripts (standalone parsers)

Related projects

Events parsers for in-world search:

w4os Web interface for OpenSimulator WordPress plugin for OpenSim grid management, providing also a collection of tools and helpers, including 2do services, It uses 2do.directory (see below) as default events source.
Flexible Helper Scripts standalone collection of helpers included in w4os, without web management interface, including in-world search engine, currency, events, offline messaging. It uses 2do.directory as default events source.

Public calendars to use without installing this app:

2do.directory, the public 2do Events hypergrid directory.
OutWorldz OpensimEvents another calendar based on HYPEvents/2do Events.

Recommended usage

The easiest and more efficient way to enable events search in your grid is to use 2do Directory, the public calendar based on this application. It is the recommended way for hypergrid-enabled regions and grid, as not only it avoids the installation of additional software, but it also allows to share events between grids.

The directory provides places, land for sale and events search in-world from the viewer search feature. Places and land for sale are parsed regularly from your simulator(s).

Enable in-world search on your region(s)

To enable in-world search in your region without installing the web application:

Download OpenSimSearch.Module.dll and put it inside your OpenSimulator bin/ folder.

Add the following settings to each simulator OpenSim.ini. (replace "Your Grid" and "yourgrid.org:8002" with your own values)

Note: Even for grids, this config goes in regions OpenSim.ini, not in Robust.ini.

[Search]
    SearchURL = "http://2do.directory/helpers/query.php?gk=http://yourgrid.org:8002"

[DataSnapshot]
    index_sims = true
    gridname = "Your Grid"
    snapshot_cache_directory = "./DataSnapshot"
    DATA_SRV_2do = http://2do.directory/helpers/register.php
    ;; You can add multiple search engines
    ;DATA_SRV_OtherEngine = http://example.org/helpers/register.py

Restart the simulators

Add yoor events calendar to 2do Directory

Send us an email at dev@2do.pm, with a link to the grid's event calendar! The easier format is iCal, but we can also build a custom scraper and parser for web-based event calendars.

2DO Directory lists only events on hypergrid-enabled grids.

Put a 2DO teleport board in your region(s)

The in-world 2D0 board can be found in Speculoos Lab region: speculoos.world:8002/Lab. It provides a beautiful list of ongoing and upcoming events, with ability to click to teleport to the event location.

You can also find the latest code on GitHub GuduleLapointe/2do-board

Standalone usage

If your grid is not hypergrid-enabled or if you want to provide your own calendar, you can install the software on your servers, composed by

2DO aggregator the parser, formatting calendars for use by in-world engines
A web helper required by OpenSimulator
- Flexible Helper Scripts: helpers only, no web interface
- or w4os: a WordPress plugin including the helpers and a web management interface
OpenSimSearch module

Potentially Obsolete Questions

This is the initial project FAQ, it needs a review and some information could be outdated, but you might find some answers too. Once the transition from Python PHP is complete, I will check that.

How do you get the information for the listed events?

2DO events pulls information from the event calendars that are published on the listed grids websites. When a grid publishes an ical feed, it is easy. But many grids have web-based event calendars. We parse those (caching sub-pages to limit load on the grid's webservers) and convert them to ical.

To make it easy to see what's going on right now and provide easy access, 2DO events runs algorithms to extract hypergrid url's and times from the event calendars. This is not always easy, as various grids have different ways of indicating the location of the event. Sometimes the information is in the description, sometimes it is just a region name, sometimes it is a hypergrid url.

What can a grid or event organizer do to make sure the events are listed correctly?

First of all, make sure the source data (event calendar, website) is correct. It helps if the information is easily parseable by a computer program (see below).

Here are some guidelines to ensure most accurate listings:

Put the location information in the location field of your calendar software. 2DO events prefers hgurls (of the form 'hg.url.com:8002:Region Name'), but if you list local region names (of the form 'Region Name') 2DO events will be able to tag on the 'hg.url.com:8002:' part itself.
Use a consistent form for the location information. For example, if you use local region names, make sure they are real region names and not something like 'The far end of the most beautiful Foobar region' but just 'Foobar'. If you use hop url's, use them consistently; don't mix 'Foobar/123/43/22' with 'Foobar (123,43,22)'.
Be consistent with timezones. 2DO events will parse times and timezones correctly, and represent the event times correctly in whatever timezone the user of 2DO events selects. But if you set the timezone of your calendar to, say, EDT (US/New York) but then put times in it that are grid times, you will confuse the parser and your events will likely be listed incorrectly. Don't do that (it will confuse your users as well!). Set the calendar to 'US/Pacific' (which is grid time) and then put in the events in grid time.
When the event moves to some other place, make sure to update the event on your grid event calendar. And to be sure that people can find your event even if they have the old time or location, put some kind of beacon that will hand out a landmark to the new location at the old location.

What is the preferred format for event data?

It helps if the information is easily parseable by a computer program. The explanation of how to accomplish this is a bit technical. Website builders should be able to grasp what is explained below, but if you have any questions do not hesitate to ask at dev@2do.pm.

There are a number of options, in order of preference:

Machine-readable, structured calendar (ical, json, ...)

Parsing event data works best when it is available in a structured, machine-readable format. The de-facto format for publishing calendar information on the internet is iCalendar (.ics). Many websites with event-support can generate an ical feed. Another easily parseable format is JSON. JSON is often used to convey data from a web server to a browser, but can also be used for sharing information in a structured way to third-party applications.

An easy way to create an iCalendar is to use Google Calendar. Google calendars can be easily embedded on existing websites, and have an option to export an iCal feed.
Machine-optimized human-readable websites

If, for some reason, an ical feed or json provider is not an option (or there is already a website and you don't want to abandon that), there are some tips to make a website more easily parseable by the 2DO events algorithms:
- Don't put all the event info in one paragraph that is not structured with html tags, that makes it really hard to parse the info accurately and correctly.
- Use elements (div, span, h1, h2, ...) to structure information. For example, put the title of the event in it's own element, put the start date in an element, etc..
- Annotate the html tags to make them easily identifiable by computers. You can do this by adding an id on the tag, or a class. Id's work best for big elements, for example a main div that surrounds all the events, or the table that surrounds all the events. Classes work well for repeating elements, such as titles, times, descriptions, etc..

If you have a web-designer who makes your website, pass along the above info and he/she should be able to structure the information for you.

Unstructured webpages

While it is possible to parse an unstructered webpage (where the event info is all in one element for example), it usually doesn't work well and breaks easily. For example, some grids have a completely free-form text-field to give all the event info with start and end times written in different forms. The hgurl will be part of the text, and sometimes is just a region name or a specific club without even listing the region name. 2DO events is unable to parse these pages.

Can I add categories to my event?

Yes, you can. Just include a tag for the form '[[Category:Example]]' somewhere in the event description. Currently the following categories will be recognized:

[[Category:Music]] - music (live performance, DJ's)
[[Category:Lecture]] - Lecture or reading
[[Category:Social]] - Social events (community meetings and such)

More categories will be added in the future. If there is a category you are missing, let us know at dev@2do.pm.

What grids are listed?

The following calendars are imported. The list is not exhaustive.

(list will be update soon and/or available elsewhere)

I don't want you to list my events!

That's a pity, but it's your call! Just let us know and we'll work out how to exclude your events.

Changelog

0.1.6

added fetcher cache to ease the developer's life
fix mobile flex style
fix mobile flex style
removed the barely working disclaimer, transition is finished and it works if not perfectly, at least better than before
set cache timeout to 55 minutes, to accomodate for 1 hour cron job
updated fluid calendar style
removed disclaimer, scraping engine is globally back to normal
set stable to 0.1.6

0.1.5

new exclusion list
added opensimworld crawler
added color differentiation for events happening today on the user time zone
added dark theme
fix undefined $source_tz
fix date and week display according to selected timezone
fix wrong date in html day block title
fix section menu disabling timezone selector
changed menu, set calendar first
removed deprecated fabpot/goutte package, use Symfony BrowserKit and DomCrawler instead
ignore composer.lock
Merge branch 'master' of github.com:GuduleLapointe/2do-aggregator
removed obsolete fetcher.cfg.example
revert bug introduced in 9c0771cc93b74c2d54188d052db2b68428852bbf
balance the layout by adjusting day columns width based on the number of events
remove rss link until it is implemented
removed debug output
detect event hgurl from description if location is not set
updated styles header padding
updated FAQ

0.1.4

added sections About (readme), FAQ and Changelog to web page
added menu
fix responsive display and text wrapping for small devices
better checkbox display in lists
use minified json, css and js

0.1.3

pretty functional version
import iCal (.ics) calendars
export json (for events parser helper and web page)
export hypevents format (extra light for 2do boards and lsl scripts)
export iCal (.ics) for use with any calendar software
export HTML, a basic web page with the calendar, with adjustable timezone
responsive layout

0.1.3-dev-3

added disclaimer to the web page
added teleport links on the web page
added real time SLTime clock
added timezone selector
added disclaimer to the web page
added teleport links on the web page
added real time SLTime clock
added basic HTML export
added timezone selector
fix new lines in description
renamed events.js as script.js
sticky header
updated disclaimer
responive events block display grouped by week and day

0.1.3-dev-2

added json format export
added matthiasmullie/minify library
fix wrong type for empty tags
fix iconv(): Detected an illegal character in input string
fix wrong end date in json export
fix $array typo
fix version
include saved files in output
don't include source in event tags
disambiguation, renamed categories as tags, to differientiate from OS/SL events category
mv src/2do-server src/helpers dev/
don't encode to utf8 if it is already the good format
minified json