Filter URLs with AI-powered robots.txt compliance & source verification
Schedule Trigger
Get Base URL
Prepare Robots.txt Check
Assume Disallow
Information Extractor
openai
Model Selector
modelselector
Mistral Cloud Chat Model
openai
Groq Chat Model
openai
Google Gemini Chat Model
google
Get Robots.txt
Check Robots.txt
Prepare Robots.txt Check 2
If Link Allowed
Prepare Output for Link Disallowed
If Link Allowed 2
Create robots.txt Table
postgres
Check robots.txt Table
postgres
If robots.txt Found and Updated
Output
Prepare Output
Prepare Output for Link Allowed
Upsert robots.txt Table
postgres
Delete robots.txt Table
postgres
Start
Check all robots.txt Table
postgres
Assume False Output
Create forbidden urls table
postgres
Check forbidden urls table
postgres
If forbidden url Detected
Return Disallow
Check all forbidden urls Table
postgres
Delete forbidden urls Table
postgres
Manual Upsert robots.txt Table
postgres
Manual Upsert forbidden urls Table
postgres
Start (Tests)
Manual Create forbidden urls table
postgres
Manually Create robots.txt Table
postgres
Press enter or space to select a node. You can then use the arrow keys to move the node around. Press delete to remove it and escape to cancel.
Press enter or space to select an edge. You can then press delete to remove it or escape to cancel.