A. Why should I run my own Mobilemaps?
- Why should I run my own Mobilemaps?
The Mobilemaps nearby engine will help users in your local city/state/country save time finding local information, but what's in it for you?
- it's fun - run some cool, cutting edge technology
- it's likely to be profitable - local search advertising is probably the "best targeted" (most relevant) advertising yet on the Web, indicating it could be the most profitable. Some local advertising analysts have predicted existing Yellow Pages revenues could move in this direction.
- it helps the Web community - many people think it's important the Web remains open and not dominated by monopolies.
If you represent a company such as a Yellow Pages or local newspaper, then by using open source Mobilemaps you:
- head off competition from search companies moving into your market
- avoid lock-in to a single service provider for technology and advertising
- avoid development overhead and software licensing fees
If you are not yet ready to run your own Mobilemaps software, remember that you will usually be able to link to someone elses, and there are people such as the developers who are prepared to help with, or host the service you are looking for.
For further background information, there are links to published articles and press on our News page which describe Mobilemaps.
[top]
- Is it free?
Yes. The software is completely free and open source (GPL) which allows you to modify and redistribute your software. The Web data is also freely harvested from the World Wide Web. (If you wanted to use paid Yellow Pages lists or point of interest data, you could use that also. We think it would be a mistake to only use that, because Web search may be the future here, just as it has succeeded in traditional search.)
However, the map and positioning/geocoding data is only free in a limited number of countries, such as the USA. European countries, on the other hand, typically have expensive licenses covering their geographic data. There are sometimes ways to minimise costs, particularly if you are prepared to use lower resolution international data such as the "Digital Chart of the World" - or intend to cover only small areas in more detail, where satellite imagery or aerial photography could prove a cheap replacement.
You will need access to one or more computers and corresponding bandwidth. One co-located computer might be fine for a US state, and a shared account (offering MySQL and perl scripting) might be suitable for a single city. Suppose you're looking to do a country like America as a whole, you will need a number of hard-drives/computers.
[top]
- Is Mobilemaps software restricted in the geographic areas it covers?
No - in that the underlying search technology is designed for operation anywhere*. But, the current Mobilemaps developers only have direct experience with geo-coding Web pages from the USA and UK. The default Mobilemaps configuration is recognition of street addresses in the USA using TIGER census data, and recognition of postcodes in the UK (UK addresses can be matched against an address database for more accurate geocoding - the developers have done this in the past). Other places will take individual work, which we hope the open source development community will assist with. The two typical approaches are to recognise addresses on pages - helped usually by a quickly recognisable postcode pattern, or to recognise phone numbers and then use a white-pages directory to get addresses. In both cases you usually need a table of addresses to geographic coordinates.
There is a generic solution to world-wide geo-coding of Web pages which we'll be re-launching shortly, based on Web masters manually placing meta-tags in Web pages that describe the geographic location of the page (obtained from a GPS unit or looking at a map). This will be available at gstart.com but it may take some time to build a useful volume of data.
* Except that Mobilemaps has not been tested with polar regions, which traditionally cause trouble with spatial software.
[top]
- How much experience do I need?
Ultimately we hope the answer becomes "not much", but currently there are some rough edges which require at least the willingness to learn/dabble, and favour some experience.
The installation relies on a range of open source software (MySQL, Perl, Apache, MapServer) which must be installed prior to Mobilmaps, and some experience with these packages already is beneficial, although we try to guide you through their installation in the INSTALL files included with the software. The Mobilemaps installation itself requires a working knowledge of your operating system, and Web sites. Currently our installation document is biased in favour of Linux which we use ourselves, and while other Unix and MacOS-X users share few differences, Windows users may have to read between the lines somewhat. Generating the maps still requires some perseverence, and certainly any geographic experience will be useful.
Support options are covered in the installation section below
[top]
- How do I contribute to the development?
We welcome contributions from developers on all levels. There are opportunities to help on everything from documentation and installation, to relevance algorithms, geocoding, and map presentation. Here is a document that briefly describes our R&D direction and where you could help.
We are starting to make use of our account on sourceforge for various aspects of our development, so developers should keep watch of our pages there.
Existing providers of map and geocoding data could investigate offering their services to Mobilemaps operators for a percentage of advertising revenue.
[top]
- How do I fund/sponsor/donate to the project?
We are in great need of computers and bandwidth, and would also welcome contact from possible investors, particularly corporations who would like to see a level playing field in local search, rather than monopoly domination. We perform any commercial operations through High Country Software Ltd, our UK based private limited company. Our contact details are here.
[top]
B. How do I run my own Mobilemaps?
- How do I download Mobilemaps?
The Mobilemaps download page is here. Currently you must download three packages: lbs, spider, and placenames. You will need all three to properly configure Mobilemaps. The lbs package is the search engine, indexer, geocoder, and also includes sox, the single box input; the spider package is the Web robot/crawler that recognises addresses on pages and intelligently associates addresses with other pages; placenames is a useful list of placenames with corresponding geographic coordinates.
In the installation section below you will discover that this is not all you will need to download. Map data, address geocoding data, and possibly the open directory project are all large downloads. For the state of California, the maps and geocoding data were each approximately 1G.
[top]
- How do I install Mobilemaps?
Installation documents for the different components that make up Mobilemaps are below:
There are a number of sources of support. Free support in the form of newsgroups are available for MySQL, Perl, Apache, MapServer; and for Mobilemaps on our site. The Mobilemaps developers also offer paid support.
If you are interested in getting the basics working quickly you could focus on the spider and lbs, and leave the mapping and geocoding for later. Basic geocoding for US addresses can be accomplished simply with zipcodes as a test (not very accurate however). The mapserver does not have to be running to see the Web pages plotted in their geographic locations, (because lbs uses HTML CSS2 code to plot them). Also sometimes it is simpler to start testing the mapserver with "small scale" global maps than to try the more particular street map datasets.
[top]
- What is the architecture of Mobilemaps?
The Mobilemaps architecture is introduced here as either a starting point for developers, or an aid for administrators wishing to understand the system better:
Mobilemaps is a system that finds textually indexed information (typically Web pages) based on geographic location.
spider is a Web crawler/harvester designed to find geographically relevant Web pages on the World Wide Web. MMSpider.pl is the graphical user interface, and MMSpider_cmdline_start.pl is the command line script. spider.pl is an individual spider process that is forked a variable number of times by the user, with each process running in parallel. The spider finds US or UK addresses on Web pages, and stores them in its database for later "geocoding" (geographically locating with a coordinate). One of the novel features of the spider is to associate physical addresses on Web pages with referrer pages, even though those pages do not contain an address themselves. For example a contacts page will often determine the geographic location of a home page for a Web site.
The spider package also includes relevance.pl the program used to calculate the relevance of each word on each Web page. Relevance in Mobilemaps is a combination of traditional search relevance and proximity. Development of the relevance.pl program will be ongoing, and a number of different options may be available in future. Diversity amongst relevance algorithms here will ultimately help to minimise the success of search engine spam against Mobilemaps. The traditional aspect of relevance can be less demanding than with traditional search, because proximity also plays a significant role. What works with traditional search, however, often does not work with location search, and a number of new relevance methods are likely to be introduced by ourselves and others.
lbs stands for "location based search" and is the core of the Mobilemaps nearby-engine. It is a combination of a keyword and spatial search index. To gain a developer's understanding of a traditional search index, the following article is quite useful (sadly it wasn't around when we wrote lbs). The spatial component of Mobilemaps relies on using a pair of space curves, based on what are called "peano codes", to simulate proximity. This allows the relevance, based in part on proximity to be calculated ahead of time, which is the prerequisite for creating a scalable search. A lack of this technology has limited previous attempts at local searches. Mobilemaps may currently be unique in providing a fully scalable spatial-keyword index that can search an arbitrary sized database, and does not limit the search to a set geographic area - like for example a 15 mile radius. lbs is the name of the package and also the name of the Web script that is run to perform a search. Markers_s.pm is the actual search module that does the real work of accessing the index in the database. Unlike many search engines lbs finds advertising results in the same search lookup as ordinary results (they are clearly differentiated from actual search results in the user presentation however!)
A Mapserver, such as the open source MapServer software, is typically used alongside lbs to provide a map of the current search area. The lbs code was originally developed to cater for other types of map systems such as static 2D maps, static 3D maps, and even interactive 3D maps, but these are not currently maintained, and may be better performed if necessary via the future XML output of search results. lbs provides an interactive user interface with pans and zooms onto the map view.
mm_index_us.pl, also included in the lbs package, is the current program used to create the keyword-spatial index. It goes through each word of each Web page in turn and places it in a reverse index that can be looked up by entering a word and a location.
A geocoding module, Usgeocode.pm is used by the indexer, mm_index_us.pl, to identify the geographic location for each address found by the spider. It relies on the creation of a geocoding database as detailed here. This will need further coding for countries outside the United States. In the UK for example, we used a slightly different approach, because we had access to a complete address table for the country which allowed a direct lookup of address to coordinates. It is also worth noting Mobilemaps has a policy of only finding one location per Web page, which we feel will ultimately benefit local search.
An additional component mm_grobot.pl has two functions: it will be used in future as part of the distributed spider/geographic internet initiative being developed; but currently it is part of the advertising solution for those people choosing to use NearbyAds. It regularly checks the mobilemaps.com site for new NearbyAds and downloads and indexes any relevant ones on the fly.
[top]
- How do I run Mobilemaps?
After installing Mobilemaps the steps to run it are as follows:
- Spider the Web in your local area:
- Find a "seed" of Web pages. This could be a section of the Open Directory Project urls (download the file from dmoz.org and use read_odp.pl for this purpose), or for example selecting a local chamber of commerce url as a single starting point - whatever you find useful. In future, when the collaborative spider functionality has been finished, this step will be easier and will just involve automatically connecting to a central server and downloading a seed of urls relevant to your local area.
- Import the seed, use the "file->import menu option" in the GUI, MMSpider.pl, or use the command-line import.pl script.
- Start the spider using either the GUI, MMSpider.pl, or the MMSpider_cmdline_start.pl command line interface. With the command line interface you can modify the configuration by updating the spider_options table in a mysql client.
- Run the relevance.pl script to calculate the relevance:
perl relevance.pl
- Export the spidered data, using the "file->export" option in the GUI, MMSpider.pl, or on the command line running the export.pl script:
perl export.pl
(In future the export and indexing processes will be combined to avoid the need to create a large text file export.)
- Index the spidered data (prepare to wait - this can be slow with large quantities of data), after which lbs will be ready to go:
perl mm_index_us.pl
- To refresh the data, you must follow the same spider and indexing steps above, except you can save time during the spidering phase by using the following options:
- Rewind the spider, by pressing the Rewind button in the GUI, or on the command line entering the following in the mysql client:
mysql> UPDATE spider_options SET rewind_all_spiders = 1; UPDATE spider_url SET visits = 0;
- Restrict to the existing spidered sites rather than find new ones by ticking: "Keep to input sites" in the GUI, or on command line entering the following in the mysql client:
mysql> UPDATE spider_options SET restrict_domains = 1;
[top]
- How do I set up advertising?
Mobilemaps offers free signup to its NearbyAds, which form a network of geographically targeted, text-based advertisements that are paid for by advertisers on a cost per click basis. Mobilemaps aims to take a small percentage of the advertising revenue from each click, and distribute the remaining revenue between the different parties involved: the Portal hosting the Mobilemaps software, a potentially different Portal that provided the Web user, and a potentially different Portal that provided the advertiser who is paying for this particular NearbyAd.
Further information is available in the Portals FAQ.
- You can sign up for free here.
- Portals' Configuration (You run your own Mobilemaps nearby engine software.):
During installation of your Mobilemaps nearby engine, you will have edited a mobilemaps.conf file (having first created it from mobilemaps.conf.sample) to set up various configuration parameters.
To ensure that advertising clicks are credited to your account, you should add your Advertising ID to your mobilemaps.conf as follows: (e.g. if your Advertising ID = 12345)
<advertising_id>12345</advertising_id>
And check that the advertising flag is turned on, which looks like:
<advertising_flag>1</advertising_flag>
You should also start the gstart-robot daemon, mm_grobot.pl, which polls the list of nearby adverts regularly to keep your copy of them up to date:
$ cd [your web root]/cgi-bin/lbs/
$ ./mm_grobot.pl
[top]
