Home / CommonGrid / CommonGrid Specification
CommonGrid
Powering Open Data, Together
CommonGrid is an open-source, AI-powered platform for distributed web crawling, intelligent data extraction, dataset federation, and structured data distribution. Built with a modular architecture, it serves as a foundation for developers to build scalable open-source software stacks without recreating the underlying data collection and processing infrastructure.
Vision
CommonGrid provides a community-owned platform for discovering, extracting, organizing, and distributing structured web data. Its modular architecture allows organizations, developers, researchers, and communities to deploy only the components they need while remaining interoperable with the broader ecosystem.
The platform is designed to scale from a single workstation to globally distributed deployments and includes an expansion path toward future planet-scale infrastructure.
Core Principles
- Open Source
- Modular Architecture
- AI-Assisted Automation
- Distributed by Design
- Horizontal Scalability
- Data Portability
- Community Ownership
- Vendor Neutral
- Privacy Respecting
- API First
Core Modules
Crawl Network
Provides distributed crawling infrastructure.
Features
- Distributed crawler nodes
- Crawl scheduling
- Crawl queue management
- Incremental crawling
- Change detection
- Intelligent crawl prioritization
- Rate limiting
- Domain management
- Regional crawler deployment
- Edge crawler support
- Crawl health monitoring
- Crawl statistics
- Autonomous crawl balancing
AI Extraction Engine
Converts web content into structured data.
Features
- Automatic schema detection
- DOM analysis
- AI extraction
- Field recognition
- Layout analysis
- Pattern recognition
- Automatic parser generation
- Self-healing extraction
- Selector repair
- Schema inference
- Automatic schema evolution
- Duplicate detection
- Entity extraction
Dataset Engine
Stores and manages structured datasets.
Features
- Dataset registry
- Dataset versioning
- Historical archives
- Dataset snapshots
- Dataset metadata
- Dataset validation
- Dataset quality scoring
- Dataset lifecycle management
- Dataset compression
- Dataset export
- Dataset import
Federation Engine
Coordinates multiple CommonGrid deployments.
Features
- Federated datasets
- Dataset synchronization
- Dataset replication
- Node discovery
- Node registration
- Global dataset registry
- Regional mirrors
- Distributed ownership
- Network synchronization
Query Engine
Provides structured search.
Features
- Structured search
- Dataset filtering
- Cross-dataset search
- Geographic search
- AI-assisted querying
- Natural language search
- Query optimization
- Query history
- Saved searches
- Search analytics
API Gateway
Provides standardized access to CommonGrid.
Features
- REST APIs
- GraphQL APIs
- Dataset APIs
- Authentication
- Authorization
- API versioning
- Rate limiting
- API documentation
- SDK support
Data Quality Engine
Maintains dataset integrity.
Features
- Data validation
- Duplicate detection
- Missing field detection
- Anomaly detection
- Consistency checking
- Confidence scoring
- Quality reporting
- Integrity verification
AI Services
Provides intelligence throughout the platform.
Features
- AI crawler generation
- AI extraction
- AI schema generation
- AI categorization
- AI enrichment
- AI recommendations
- AI query assistant
- AI anomaly detection
- AI optimization
Knowledge Graph Engine
Builds relationships between datasets.
Features
- Entity linking
- Relationship mapping
- Knowledge graph generation
- Semantic enrichment
- Cross-reference generation
- Graph traversal
- Graph analytics
Mobility Engine
Ensures deployment portability.
Features
- Configuration portability
- Pipeline portability
- Dataset portability
- Migration tools
- Backup management
- Restore management
- Infrastructure abstraction
Monitoring & Operations
Monitors the health of the platform.
Features
- Service monitoring
- Crawl monitoring
- Dataset monitoring
- Node monitoring
- Alerting
- Metrics
- Logging
- Health dashboards
- Performance analytics
Optional Plugin Modules
Plugins extend CommonGrid without modifying the core platform.
Autonomous Discovery
- Automatic website discovery
- Dataset discovery
- Sitemap discovery
- Domain intelligence
Historical Intelligence
- Timeline analysis
- Historical comparisons
- Trend detection
- Change visualization
Knowledge Marketplace
- Dataset publishing
- Dataset subscriptions
- Dataset catalogs
- Community sharing
Multi-Format Extraction
- PDF extraction
- Spreadsheet extraction
- XML extraction
- JSON extraction
- RSS extraction
- Document parsing
Translation & Localization
- Automatic translation
- Multilingual datasets
- Language detection
- Localization tools
Geographic Intelligence
- Geocoding
- Reverse geocoding
- Geographic enrichment
- Spatial analysis
- Mapping support
Machine Learning
- Custom model deployment
- Model training
- Prediction services
- Classification
- Recommendation models
Workflow Automation
- Scheduled workflows
- Event automation
- Pipeline automation
- Task orchestration
Notifications
- Email notifications
- Webhooks
- Push notifications
- Event alerts
- Status updates
Security
- Secret management
- Audit logging
- Role management
- Access policies
- Encryption services
Identity
- Single Sign-On
- OAuth
- OpenID Connect
- LDAP
- Multi-factor authentication
Analytics
- Usage analytics
- Crawl analytics
- Dataset analytics
- API analytics
- Network analytics
Developer Tools
- CLI tools
- SDK generation
- API testing
- Development utilities
- Debugging tools
Visualization
- Dashboards
- Dataset explorer
- Graph visualization
- Maps
- Charts
- Reports
Backup & Recovery
- Scheduled backups
- Disaster recovery
- Point-in-time recovery
- Archive management
Planet-Scale Expansion
Future expansion modules.
Features
- Continental crawl clusters
- Global dataset routing
- Autonomous infrastructure optimization
- Intelligent workload distribution
- Global replication
- Planet-scale indexing
- Massive knowledge graph federation
- Autonomous network management
Roadmap
Version 0.1
- Local deployment
- Distributed crawler
- AI extraction
- Dataset engine
Version 0.5
- Federation
- Query engine
- API Gateway
- AI services
Version 1.0
- Global deployment
- Knowledge graph
- Mobility engine
- Monitoring
Future
- Planet-scale expansion
- Autonomous discovery
- Intelligent optimization
- Community dataset ecosystem
Specification Branding License (SBL)
Standard
- Fully AGPL-3.0+ compliant system
- Copyleft enforced for network deployments
- Required attribution:
- Roxanne Ardary
- https://www.roxanneardary.com/
Optional
- Specification Branding License (SBL)
- Attribution-free commercial deployment
- Pricing based on scale, usage, and deployment scope
- https://roxanneardary.com/commongrid/
License & Notice Requirements
CommonGrid is released under the GNU Affero General Public License v3.0 or later (AGPL-3.0+).
By contributing to this project, you agree that your contributions will also be released under this license.
Please note the following:
- All contributions must comply with the AGPL-3.0+ terms.
- Under Section 7 of the license, all redistributions, forks, and derivative works must preserve attribution to:
Roxanne Ardary and roxanneardary.com. - CommonGrid specifications are free to use with attribution. A Specification Branding License can be negotiated upon request.
- The project’s notice.md file tracks attribution requirements and contributor acknowledgments. Any update that adds new contributors or modifies attribution should also update
notice.md. - When submitting a pull request, ensure that any new files maintain the attribution headers where applicable.
- Network-deployed versions of this software must also remain fully AGPL-3.0+ compliant, including exposure of source code modifications when applicable under the license.
For full legal details, please refer to the AGPL-3.0+ license and the project’s notice.md file.
Notice – CommonGrid
Attribution Requirement: Under Section 7 of the AGPL 3.0+ license, all redistributions, forks, and derivative works, including network-deployed versions of this project, must provide attribution to Roxanne Ardary and roxanneardary.com.
Contributors
This file tracks contributors and their specific contributions to the project.
- Roxanne Ardary, roxanneardary.com – March 12, 2026
Created the repository for CommonGrid. Designed the AI-powered, decentralized, global data extraction and distribution platform. - [Add other contributors here] – [Date]
[Describe contribution in one sentence]
License – CommonGrid
This repository is licensed under the GNU Affero General Public License v3.0 or later (AGPL-3.0+).
Key Points
- You are free to use, modify, and distribute the code.
- All redistributions, forks, and derivative works or network-deployed versions must also be licensed under AGPL-3.0+ and provide attribution to Roxanne Ardary and roxanneardary.com as required under Section 7 of the license.
- The software is provided “as is,” without warranty of any kind.
For the full license text, see GNU AGPL-3.0 License.