# Google # UA "AdsBot-Google (+http://www.google.com/adsbot.html)" # UA "Googlebot-Image/1.0" # UA "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" # UA "Googlebot/Test (+http://www.googlebot.com/bot.html)" # UA "Googlebot/Test" # UA "Mediapartners-Google/2.1 (+http://www.googlebot.com/bot.html)" # UA "Mediapartners-Google/2.1" # UA "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" # UA "gsa-crawler (Enterprise; S4-E9LJ2B82FJJAA; me@mycompany.com)" 209.185.108 209.185.253 216.239.33.96 216.239.33.97 216.239.33.98 216.239.33.99 216.239.37.98 216.239.37.99 216.239.39.98 216.239.39.99 216.239.41.96 216.239.41.97 216.239.41.98 216.239.41.99 216.239.45.4 216.239.46 216.239.51.96 216.239.51.97 216.239.51.98 216.239.51.99 216.239.53.98 216.239.53.99 216.239.57.96 216.239.57.97 216.239.57.98 216.239.57.99 216.239.59.98 216.239.59.99 216.33.229.163 64.233.173.193 64.233.173.194 64.233.173.195 64.233.173.196 64.233.173.197 64.233.173.198 64.233.173.199 64.233.173.200 64.233.173.201 64.233.173.202 64.233.173.203 64.233.173.204 64.233.173.205 64.233.173.206 64.233.173.207 64.233.173.208 64.233.173.209 64.233.173.210 64.233.173.211 64.233.173.212 64.233.173.213 64.233.173.214 64.233.173.215 64.233.173.216 64.233.173.217 64.233.173.218 64.233.173.219 64.233.173.220 64.233.173.221 64.233.173.222 64.233.173.223 64.233.173.224 64.233.173.225 64.233.173.226 64.233.173.227 64.233.173.228 64.233.173.229 64.233.173.230 64.233.173.231 64.233.173.232 64.233.173.233 64.233.173.234 64.233.173.235 64.233.173.236 64.233.173.237 64.233.173.238 64.233.173.239 64.233.173.240 64.233.173.241 64.233.173.242 64.233.173.243 64.233.173.244 64.233.173.245 64.233.173.246 64.233.173.247 64.233.173.248 64.233.173.249 64.233.173.250 64.233.173.251 64.233.173.252 64.233.173.253 64.233.173.254 64.233.173.255 64.68.80 64.68.81 64.68.82 64.68.83 64.68.84 64.68.85 64.68.86 64.68.87 64.68.88 64.68.89 64.68.90.1 64.68.90.10 64.68.90.11 64.68.90.12 64.68.90.129 64.68.90.13 64.68.90.130 64.68.90.131 64.68.90.132 64.68.90.133 64.68.90.134 64.68.90.135 64.68.90.136 64.68.90.137 64.68.90.138 64.68.90.139 64.68.90.14 64.68.90.140 64.68.90.141 64.68.90.142 64.68.90.143 64.68.90.144 64.68.90.145 64.68.90.146 64.68.90.147 64.68.90.148 64.68.90.149 64.68.90.15 64.68.90.150 64.68.90.151 64.68.90.152 64.68.90.153 64.68.90.154 64.68.90.155 64.68.90.156 64.68.90.157 64.68.90.158 64.68.90.159 64.68.90.16 64.68.90.160 64.68.90.161 64.68.90.162 64.68.90.163 64.68.90.164 64.68.90.165 64.68.90.166 64.68.90.167 64.68.90.168 64.68.90.169 64.68.90.17 64.68.90.170 64.68.90.171 64.68.90.172 64.68.90.173 64.68.90.174 64.68.90.175 64.68.90.176 64.68.90.177 64.68.90.178 64.68.90.179 64.68.90.18 64.68.90.180 64.68.90.181 64.68.90.182 64.68.90.183 64.68.90.184 64.68.90.185 64.68.90.186 64.68.90.187 64.68.90.188 64.68.90.189 64.68.90.19 64.68.90.190 64.68.90.191 64.68.90.192 64.68.90.193 64.68.90.194 64.68.90.195 64.68.90.196 64.68.90.197 64.68.90.198 64.68.90.199 64.68.90.2 64.68.90.20 64.68.90.200 64.68.90.201 64.68.90.202 64.68.90.203 64.68.90.204 64.68.90.205 64.68.90.206 64.68.90.207 64.68.90.208 64.68.90.21 64.68.90.22 64.68.90.23 64.68.90.24 64.68.90.25 64.68.90.26 64.68.90.27 64.68.90.28 64.68.90.29 64.68.90.3 64.68.90.30 64.68.90.31 64.68.90.32 64.68.90.33 64.68.90.34 64.68.90.35 64.68.90.36 64.68.90.37 64.68.90.38 64.68.90.39 64.68.90.4 64.68.90.40 64.68.90.41 64.68.90.42 64.68.90.43 64.68.90.44 64.68.90.45 64.68.90.46 64.68.90.47 64.68.90.48 64.68.90.49 64.68.90.5 64.68.90.50 64.68.90.51 64.68.90.52 64.68.90.53 64.68.90.54 64.68.90.55 64.68.90.56 64.68.90.57 64.68.90.58 64.68.90.59 64.68.90.6 64.68.90.60 64.68.90.61 64.68.90.62 64.68.90.63 64.68.90.64 64.68.90.65 64.68.90.66 64.68.90.67 64.68.90.68 64.68.90.69 64.68.90.7 64.68.90.70 64.68.90.71 64.68.90.72 64.68.90.73 64.68.90.74 64.68.90.75 64.68.90.76 64.68.90.77 64.68.90.78 64.68.90.79 64.68.90.8 64.68.90.80 64.68.90.9 64.68.91 64.68.92 66.249.64 66.249.65 66.249.66 66.249.67 66.249.68 66.249.69 66.249.70 66.249.71 66.249.72 66.249.78 66.249.79 72.14.199 8.6.48 # Inktomi # UA "Fast Crawler v X" # UA "Fast Crawler v X(compatible; Konqueror/3.2; FreeBSD) (KHTML, like Gecko)" # UA "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)" # UA "Mozilla/4.0" # UA "Mozilla/4.05 [en]" # UA "Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html)" # UA "Mozilla/5.0 (compatible; Yahoo! DE Slurp; http://help.yahoo.com/help/us/ysearch/slurp)" # UA "Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)" # UA "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)" # UA "Scooter/3.3" # UA "Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/blog-search/)" # UA "Yahoo! Mindset" # UA "Yahoo-Blogs/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/ysearc...awling-02.html )" # UA "Yahoo-MMAudVid/1.0 (mms dash mmaudvidcrawler dash support at yahoo dash inc dot com)" # UA "Yahoo-MMCrawler/3.x (mms dash mmcrawler dash support at yahoo dash inc dot com)" # UA "YahooFeedSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; my.yahoo.com/s/publishers.html)" # UA "YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/)" # UA "YahooSeeker/1.1 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/shop/merchant/)" # UA "YahooSeeker/1.2 (compatible; Mozilla 4.0; MSIE 5.5; yahooseeker at yahoo-inc dot com ; http://help.yahoo.com/help/us/shop/merchant/)" # UA "YahooSeeker/CafeKelsa-dev (compatible; Konqueror/3.2; FreeBSD ;cafekelsa-dev-webmaster@yahoo-inc.com ) (KHTML, like Gecko)" # UA "YahooVideoSearch www.yahoo.com/" # UA "YahooYSMcm/2.0.0" # UA "slurp" 141.185.209 169.207.238 199.177.18.9 202.160.178 202.160.179 202.160.180 202.160.181.183 202.160.183.182 202.160.183.217 202.160.183.218 202.160.183.219 202.160.183.220 202.160.183.235 202.160.183.239 202.160.183.245 202.160.185.174 202.165.96.142 202.165.98 202.165.99 202.212.5.30 202.212.5.32 202.212.5.33 202.212.5.34 202.212.5.35 202.212.5.36 202.212.5.37 202.212.5.38 202.212.5.39 202.212.5.47 202.212.5.48 202.46.19.93 203.123.188.2 203.141.52.41 203.141.52.42 203.141.52.43 203.141.52.44 203.141.52.45 203.141.52.46 203.141.52.47 203.255.234.102 203.255.234.103 203.255.234.105 203.255.234.106 206.190.43.125 206.190.43.81 207.126.239.224 209.1.12 209.1.13.101 209.1.13.231 209.1.13.232 209.1.32.122 209.1.38 209.131.40 209.131.41 209.131.48 209.131.49.37 209.131.50.153 209.131.51.166 209.131.60.169 209.131.60.170 209.131.60.171 209.131.60.19 209.131.62.107 209.131.62.108 209.131.62.109 209.131.62.214 209.185.122 209.185.141 209.185.143 209.191.64.227 209.191.65 209.191.65.249 209.191.65.82 209.191.82.245 209.191.82.252 209.191.83 209.191.83.220 209.191.87.215 209.191.87.216 209.191.87.217 209.191.87.218 209.191.87.219 209.191.87.220 209.191.87.221 209.191.87.222 209.191.87.223 209.67.206.126 209.67.206.127 209.67.206.133 209.73.176.128 209.73.176.129 209.73.176.133 209.73.176.134 209.73.176.136 211.14.8.240 211.169.241.21 213.216.143.37 213.216.143.38 213.216.143.39 216.109.121.70 216.109.121.71 216.109.126.131 216.109.126.133 216.109.126.137 216.109.126.138 216.109.126.139 216.109.126.141 216.109.126.143 216.109.126.145 216.109.126.146 216.109.126.147 216.109.126.150 216.109.126.152 216.109.126.157 216.109.126.158 216.109.126.159 216.109.126.160 216.109.126.161 216.136.233.164 216.145.58.219 216.155.198.60 216.155.200 216.155.202.175 216.155.202.54 216.155.204.40 216.239.193.71 216.239.193.72 216.239.193.73 216.239.193.74 216.239.193.75 216.239.193.76 216.239.193.77 216.239.193.78 216.239.193.79 216.239.193.80 216.239.193.81 216.239.193.82 216.239.193.83 216.239.193.84 216.239.193.85 216.239.193.86 216.32.237.1 216.32.237.10 216.32.237.11 216.32.237.12 216.32.237.13 216.32.237.14 216.32.237.15 216.32.237.16 216.32.237.17 216.32.237.18 216.32.237.19 216.32.237.20 216.32.237.21 216.32.237.22 216.32.237.23 216.32.237.24 216.32.237.25 216.32.237.26 216.32.237.27 216.32.237.28 216.32.237.29 216.32.237.30 216.32.237.7 216.32.237.8 216.32.237.9 62.172.199.20 62.172.199.21 62.172.199.22 62.172.199.23 62.172.199.24 62.27.59.245 63.163.102.180 63.163.102.181 63.163.102.182 64.157.137.219 64.157.137.220 64.157.137.221 64.157.137.225 64.157.138.103 64.157.138.108 64.75.36.42 64.75.36.43 64.75.36.44 64.75.36.45 64.75.36.47 64.75.36.77 64.75.36.79 64.75.36.80 66.163.170.157 66.163.170.159 66.163.170.161 66.163.170.162 66.163.170.166 66.163.170.167 66.163.170.170 66.163.170.172 66.163.170.176 66.163.170.178 66.163.170.179 66.163.170.180 66.163.170.184 66.163.170.185 66.163.170.190 66.163.170.192 66.163.174.65 66.196.101 66.196.65 66.196.67.100 66.196.67.101 66.196.67.102 66.196.67.103 66.196.67.104 66.196.67.105 66.196.67.106 66.196.67.107 66.196.67.108 66.196.67.109 66.196.67.110 66.196.67.111 66.196.67.112 66.196.67.113 66.196.67.114 66.196.67.115 66.196.67.116 66.196.67.117 66.196.67.118 66.196.67.119 66.196.67.120 66.196.67.121 66.196.67.122 66.196.67.123 66.196.67.124 66.196.67.125 66.196.67.126 66.196.67.127 66.196.67.128 66.196.67.129 66.196.67.130 66.196.67.150 66.196.67.151 66.196.67.176 66.196.67.177 66.196.67.178 66.196.67.200 66.196.67.201 66.196.67.202 66.196.67.203 66.196.67.204 66.196.67.205 66.196.67.206 66.196.67.207 66.196.67.208 66.196.67.209 66.196.67.210 66.196.67.211 66.196.67.212 66.196.67.213 66.196.67.214 66.196.67.215 66.196.67.216 66.196.67.217 66.196.67.218 66.196.67.219 66.196.67.220 66.196.67.221 66.196.67.222 66.196.67.223 66.196.67.224 66.196.67.225 66.196.67.226 66.196.67.227 66.196.67.228 66.196.67.229 66.196.67.230 66.196.67.231 66.196.67.232 66.196.67.233 66.196.67.234 66.196.67.235 66.196.67.236 66.196.67.237 66.196.67.238 66.196.67.239 66.196.67.240 66.196.67.254 66.196.67.30 66.196.67.31 66.196.67.32 66.196.67.33 66.196.67.34 66.196.67.35 66.196.67.36 66.196.67.37 66.196.67.38 66.196.67.39 66.196.67.70 66.196.67.71 66.196.67.72 66.196.67.73 66.196.67.74 66.196.67.75 66.196.67.76 66.196.67.77 66.196.67.78 66.196.67.79 66.196.67.80 66.196.67.94 66.196.67.95 66.196.67.96 66.196.67.97 66.196.67.98 66.196.67.99 66.196.72 66.196.73 66.196.74 66.196.77 66.196.78 66.196.80 66.196.81.10 66.196.81.102 66.196.81.103 66.196.81.104 66.196.81.105 66.196.81.106 66.196.81.107 66.196.81.108 66.196.81.109 66.196.81.11 66.196.81.110 66.196.81.111 66.196.81.112 66.196.81.113 66.196.81.114 66.196.81.115 66.196.81.116 66.196.81.117 66.196.81.118 66.196.81.119 66.196.81.12 66.196.81.120 66.196.81.121 66.196.81.122 66.196.81.123 66.196.81.124 66.196.81.125 66.196.81.126 66.196.81.127 66.196.81.128 66.196.81.129 66.196.81.13 66.196.81.130 66.196.81.131 66.196.81.132 66.196.81.133 66.196.81.134 66.196.81.135 66.196.81.136 66.196.81.137 66.196.81.138 66.196.81.139 66.196.81.14 66.196.81.140 66.196.81.141 66.196.81.142 66.196.81.143 66.196.81.144 66.196.81.145 66.196.81.146 66.196.81.147 66.196.81.148 66.196.81.149 66.196.81.15 66.196.81.150 66.196.81.151 66.196.81.152 66.196.81.153 66.196.81.154 66.196.81.155 66.196.81.156 66.196.81.157 66.196.81.158 66.196.81.159 66.196.81.16 66.196.81.160 66.196.81.161 66.196.81.162 66.196.81.163 66.196.81.164 66.196.81.165 66.196.81.166 66.196.81.167 66.196.81.168 66.196.81.169 66.196.81.17 66.196.81.170 66.196.81.171 66.196.81.172 66.196.81.173 66.196.81.174 66.196.81.175 66.196.81.176 66.196.81.177 66.196.81.178 66.196.81.179 66.196.81.18 66.196.81.180 66.196.81.181 66.196.81.182 66.196.81.183 66.196.81.184 66.196.81.185 66.196.81.187 66.196.81.188 66.196.81.189 66.196.81.19 66.196.81.190 66.196.81.191 66.196.81.192 66.196.81.193 66.196.81.194 66.196.81.195 66.196.81.196 66.196.81.197 66.196.81.198 66.196.81.199 66.196.81.20 66.196.81.200 66.196.81.201 66.196.81.202 66.196.81.203 66.196.81.204 66.196.81.205 66.196.81.206 66.196.81.207 66.196.81.208 66.196.81.209 66.196.81.21 66.196.81.210 66.196.81.211 66.196.81.212 66.196.81.213 66.196.81.214 66.196.81.215 66.196.81.216 66.196.81.217 66.196.81.218 66.196.81.219 66.196.81.22 66.196.81.23 66.196.81.86 66.196.81.87 66.196.81.88 66.196.81.93 66.196.81.94 66.196.81.95 66.196.81.96 66.196.90 66.196.91 66.196.92 66.196.93.19 66.196.93.24 66.196.93.6 66.196.93.7 66.196.97 66.196.99.20 66.218.65.52 66.218.70 66.228.164 66.228.165 66.228.166 66.228.173 66.228.182.177 66.228.182.183 66.228.182.185 66.228.182.187 66.228.182.188 66.228.182.190 66.94.230.100 66.94.230.101 66.94.230.102 66.94.230.103 66.94.230.104 66.94.230.105 66.94.230.106 66.94.230.107 66.94.230.108 66.94.230.109 66.94.230.110 66.94.230.160 66.94.230.161 66.94.230.162 66.94.230.163 66.94.230.96 66.94.230.97 66.94.230.98 66.94.230.99 66.94.232 66.94.233 66.94.238.51 68.142.195.80 68.142.195.81 68.142.203.133 68.142.211.69 68.142.212.197 68.142.230.125 68.142.230.126 68.142.230.127 68.142.230.128 68.142.230.129 68.142.230.130 68.142.230.131 68.142.230.132 68.142.230.133 68.142.230.134 68.142.230.135 68.142.230.136 68.142.230.137 68.142.230.138 68.142.230.139 68.142.230.140 68.142.230.141 68.142.230.142 68.142.230.143 68.142.230.144 68.142.230.145 68.142.230.146 68.142.230.147 68.142.230.148 68.142.230.149 68.142.230.150 68.142.230.151 68.142.230.152 68.142.230.153 68.142.230.154 68.142.230.155 68.142.230.156 68.142.230.157 68.142.230.158 68.142.230.159 68.142.230.160 68.142.230.161 68.142.230.162 68.142.230.163 68.142.230.164 68.142.230.165 68.142.230.166 68.142.230.167 68.142.230.168 68.142.230.169 68.142.230.174 68.142.230.175 68.142.230.176 68.142.230.177 68.142.230.178 68.142.230.179 68.142.230.180 68.142.230.181 68.142.230.182 68.142.230.183 68.142.230.184 68.142.230.185 68.142.230.186 68.142.230.187 68.142.230.188 68.142.230.189 68.142.230.190 68.142.230.191 68.142.230.192 68.142.230.193 68.142.230.194 68.142.230.195 68.142.230.196 68.142.230.197 68.142.230.198 68.142.230.199 68.142.230.200 68.142.230.201 68.142.230.202 68.142.230.203 68.142.230.204 68.142.230.205 68.142.230.206 68.142.230.207 68.142.230.208 68.142.230.209 68.142.230.210 68.142.230.211 68.142.230.212 68.142.230.213 68.142.230.214 68.142.230.215 68.142.230.216 68.142.230.217 68.142.230.240 68.142.230.247 68.142.230.248 68.142.230.249 68.142.230.250 68.142.230.251 68.142.230.252 68.142.230.253 68.142.230.254 68.142.230.32 68.142.230.33 68.142.230.34 68.142.230.35 68.142.230.36 68.142.230.37 68.142.230.38 68.142.230.39 68.142.230.40 68.142.230.41 68.142.230.43 68.142.230.44 68.142.230.45 68.142.230.46 68.142.230.47 68.142.230.48 68.142.230.49 68.142.231.49 68.142.240.106 68.142.246 68.142.249 68.142.250 68.142.251 69.147.79.131 69.147.79.137 69.147.79.173 72.30.101 72.30.102 72.30.103 72.30.104 72.30.107 72.30.110 72.30.111 72.30.124.128 72.30.124.130 72.30.124.134 72.30.128 72.30.129 72.30.131 72.30.132 72.30.133 72.30.134 72.30.135 72.30.142.24 72.30.142.25 72.30.177 72.30.214 72.30.215 72.30.216 72.30.221 72.30.226 72.30.252 72.30.54 72.30.56 72.30.60 72.30.61 72.30.81 72.30.87 72.30.9 72.30.97 72.30.98 72.30.99 74.6.131 74.6.240 74.6.65 74.6.66 74.6.67 74.6.68 74.6.69 74.6.70 74.6.71 74.6.72 74.6.73 74.6.74 74.6.75 74.6.76 74.6.85 74.6.86 74.6.87 # Lycos # UA "Lycos_Spider_(modspider)" 166.48.225.254 202.232.118.40 202.232.118.51 206.79.171 207.77.90 207.77.91.184 208.146.26 208.146.27.123 208.146.27.124 208.146.27.57 208.146.27.58 208.146.27.59 208.146.27.60 208.146.27.62 208.146.27.89 208.146.27.90 208.146.27.91 208.146.27.92 208.146.27.93 208.146.27.94 208.146.27.95 208.146.27.96 209.202.192 209.202.192.147 209.202.193 209.202.194.237 209.202.194.238 209.202.205.1 209.202.240.109 209.202.240.8 209.202.248.211 209.202.248.212 209.202.248.213 209.202.248.214 209.67.228 209.67.229 211.51.63.4 213.193.19.35 64.89.33 # UA "MSNBOT/0.1 (http://search.msn.com/msnbot.htm)" # UA "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)" # UA "lanshanbot/1.0 ( http://search.msn.com/msnbot.htm)" # UA "msnbot-Products/1.0 (+http://search.msn.com/msnbot.htm)" # UA "msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)" # UA "msnbot/0.11 (+http://search.msn.com/msnbot.htm)" # UA "msnbot/0.3 (+http://search.msn.com/msnbot.htm)" # UA "msnbot/0.9 (+http://search.msn.com/msnbot.htm)" # UA "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" 131.107.0 131.107.137.165 131.107.137.166 131.107.137.37 131.107.137.47 131.107.163.47 131.107.163.48 131.107.163.49 131.107.163.57 131.107.163.58 131.107.76.139 202.96.51.137 202.96.51.144 202.96.51.145 202.96.51.149 202.96.51.150 202.96.51.151 202.96.51.152 202.96.51.153 202.96.51.154 202.96.51.155 202.96.51.156 202.96.51.157 202.96.51.158 202.96.51.159 202.96.51.162 202.96.51.172 204.4.80.10 204.95.98.251 204.95.98.252 204.95.98.253 207.46.238.133 207.46.238.137 207.46.238.138 207.46.238.143 207.46.71.12 207.46.71.16 207.46.89.16 207.46.89.17 207.46.98 207.68.146 207.68.154 207.68.157 207.68.188.241 207.68.188.242 207.68.188.243 207.68.188.244 213.199.128.156 63.194.155.144 63.194.155.145 63.194.155.146 63.194.155.147 63.194.155.148 63.194.155.149 63.194.155.150 63.194.155.151 64.4.8 65.54.164.100 65.54.164.101 65.54.164.102 65.54.164.103 65.54.164.104 65.54.164.105 65.54.164.106 65.54.164.107 65.54.164.108 65.54.164.109 65.54.164.110 65.54.164.111 65.54.164.112 65.54.164.113 65.54.164.114 65.54.164.115 65.54.164.116 65.54.164.117 65.54.164.118 65.54.164.119 65.54.164.120 65.54.164.121 65.54.164.122 65.54.164.123 65.54.164.124 65.54.164.125 65.54.164.126 65.54.164.127 65.54.164.128 65.54.164.129 65.54.164.130 65.54.164.131 65.54.164.132 65.54.164.133 65.54.164.134 65.54.164.135 65.54.164.34 65.54.164.35 65.54.164.36 65.54.164.37 65.54.164.38 65.54.164.39 65.54.164.40 65.54.164.41 65.54.164.42 65.54.164.43 65.54.164.44 65.54.164.45 65.54.164.46 65.54.164.47 65.54.164.48 65.54.164.49 65.54.164.50 65.54.164.51 65.54.164.52 65.54.164.53 65.54.164.54 65.54.164.55 65.54.164.56 65.54.164.57 65.54.164.58 65.54.164.59 65.54.164.60 65.54.164.61 65.54.164.62 65.54.164.63 65.54.164.64 65.54.164.65 65.54.164.66 65.54.164.67 65.54.164.68 65.54.164.69 65.54.164.70 65.54.164.71 65.54.164.72 65.54.164.73 65.54.164.74 65.54.164.75 65.54.164.76 65.54.164.77 65.54.164.78 65.54.164.79 65.54.164.80 65.54.164.81 65.54.164.82 65.54.164.83 65.54.164.84 65.54.164.85 65.54.164.86 65.54.164.87 65.54.164.88 65.54.164.89 65.54.164.90 65.54.164.91 65.54.164.92 65.54.164.93 65.54.164.94 65.54.164.95 65.54.164.96 65.54.164.97 65.54.164.98 65.54.164.99 65.54.165 65.54.188 65.54.189 65.54.236.87 65.54.236.88 65.54.236.90 65.55.208 65.55.209 65.55.210 65.55.212 65.55.213 65.55.214 65.55.215 65.55.216 65.55.233 65.55.235 65.55.246 65.55.252 68.55.252 # Alta Vista # UA "Scooter/3.3Y!CrawlX" 128.177.243 128.177.244.100 128.177.244.86 194.112.94.250 194.112.94.251 194.112.94.252 194.201.146.1 194.201.146.24 194.221.84.11 194.221.84.15 194.221.84.31 194.221.84.32 194.221.84.33 194.221.84.34 194.221.84.38 194.221.84.39 194.221.84.40 194.221.84.41 194.51.33.72 204.123.13.36 204.123.13.65 204.123.13.66 204.123.2 204.123.28.10 204.123.28.11 204.123.28.20 204.123.28.21 204.123.28.27 204.123.28.30 204.123.28.31 204.123.28.32 204.123.28.33 204.123.9 204.152.190 204.152.191 205.229.83.18 208.185.243.148 208.221.32 208.221.35.200 208.221.35.201 208.221.35.202 208.221.35.203 208.221.35.204 208.221.35.205 208.221.35.206 208.221.35.207 209.247.40.246 209.73.160.1 209.73.160.250 209.73.160.254 209.73.162 209.73.164 209.73.174.250 209.73.174.251 209.73.180 212.187.213.171 212.187.213.172 212.187.213.173 212.187.213.174 212.187.213.175 212.187.226 212.187.226.42 212.187.226.83 212.187.226.84 212.187.226.85 212.187.226.87 212.187.226.88 212.187.226.93 212.187.226.99 212.187.227 216.39.48 216.39.50 216.39.51 64.152.75 66.17.148.128 66.17.148.129 66.17.148.130 66.17.148.131 66.17.148.132 66.17.148.133 66.17.148.134 66.17.148.135 66.17.148.136 66.17.148.137 66.17.148.138 66.17.148.139 66.17.148.140 66.17.148.141 66.17.148.142 66.17.148.143 66.17.148.144 66.17.148.145 66.17.148.146 66.17.148.147 66.17.148.148 66.17.148.149 66.17.148.150 66.17.148.151 66.17.148.152 66.17.148.153 66.17.148.154 66.17.148.155 66.17.148.156 66.17.148.157 66.17.148.158 66.17.148.159 66.17.148.160 66.17.148.161 66.17.148.162 66.17.148.163 66.17.148.164 66.17.148.165 66.17.148.166 66.17.148.167 66.17.148.168 66.17.148.169 66.17.148.170 66.17.148.171 66.17.148.172 66.17.148.173 66.17.148.174 66.17.148.175 66.17.148.176 66.17.148.177 66.17.148.178 66.17.148.179 66.17.148.180 66.17.148.181 66.17.148.182 66.17.148.183 66.17.148.184 66.17.148.185 66.17.148.186 66.17.148.187 66.17.148.188 66.17.148.189 66.17.148.190 66.17.148.191 66.163.170.193 # WISEnut # UA "Mozilla/4.0 compatible ZyBorg/1.0 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)" # UA "Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)" 64.241.243.123 64.241.242.177 64.241.243.65 64.241.243.124 64.242.88.10 64.242.88.50 64.242.88.60 65.116.145.141 65.113.96.174 65.116.145 66.35.208.59 66.35.208.60 66.35.208.112 66.35.208.158 66.35.208.160 66.35.208.206 66.35.208.210 66.35.208.211 208.232.154.64 209.249.66.10 209.249.66.26 209.249.66.36 209.249.67.101 209.249.67.102 209.249.67.103 209.249.67.104 209.249.67.105 209.249.67.106 209.249.67.107 209.249.67.108 209.249.67.109 209.249.67.110 209.249.67.111 209.249.67.112 209.249.67.113 209.249.67.114 209.249.67.115 209.249.67.116 209.249.67.117 209.249.67.118 209.249.67.119 209.249.67.120 209.249.67.121 209.249.67.122 209.249.67.125 209.249.67.126 209.249.67.127 209.249.67.128 209.249.67.129 209.249.67.130 209.249.67.131 209.249.67.132 209.249.67.133 209.249.67.134 209.249.67.135 209.249.67.136 209.249.67.137 209.249.67.138 209.249.67.139 209.249.67.140 209.249.67.141 209.249.67.142 209.249.67.143 209.249.67.144 210.109.141.5 216.34.42.12 216.34.42.14 216.34.42.36 216.34.42.38 216.34.42.42 216.34.42.47 216.34.42.54 216.34.42.55 216.34.42.56 216.34.42.57 216.34.42.59 216.34.42.110 216.34.42.111 216.34.42.112 216.34.42.113 216.34.42.114 216.34.42.115 216.34.42.116 216.34.42.117 216.34.42.171 216.34.42.172 216.34.42.173 216.34.42.176 216.34.42.210 216.34.42.211 216.34.42.212 216.34.42.213 216.34.42.214 216.34.42.215 216.34.42.216 216.34.42.217 216.88.158.142 # Ask Jeeves/Teoma # UA "Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html)" # UA "Mozilla/2.0 (compatible; AskJeeves/Teoma)" # UA "teoma_agent1" 140.239.126.13 140.239.251.220 140.239.251.221 140.239.251.222 140.239.251.223 140.239.251.224 140.239.251.230 207.204.132.233 207.204.132.234 208.178.104.55 209.67.252.197 209.67.252.199 209.67.252.211 209.67.252.212 209.67.252.213 209.67.252.214 209.67.252.215 209.67.252.216 210.51.25.142 211.13.230.249 216.143.191.131 216.200.130.20 216.200.130.200 216.200.130.201 216.200.130.202 216.200.130.203 216.200.130.204 216.200.130.205 216.200.130.206 216.200.130.207 216.200.130.208 216.200.130.209 216.200.130.210 216.200.130.242 216.200.130.244 216.200.130.245 216.200.130.246 216.200.130.248 216.200.130.249 216.200.130.26 216.200.130.77 216.200.130.78 216.200.130.79 216.200.130.85 216.200.130.86 216.200.130.87 216.200.130.88 216.200.130.89 216.34.121.100 216.34.121.18 216.34.121.19 216.34.121.31 216.34.121.32 216.34.121.33 216.34.121.34 216.34.121.67 216.34.121.68 63.236.92.143 63.236.92.144 63.236.92.145 63.236.92.146 63.236.92.147 63.236.92.148 63.236.92.149 63.236.92.150 63.236.92.151 63.236.92.152 63.236.92.153 64.55.148.3 64.55.148.37 64.55.148.38 64.55.148.39 64.55.148.4 64.55.148.43 64.55.148.44 64.55.148.45 64.55.148.5 64.55.148.50 64.55.148.51 64.55.148.52 64.55.148.53 64.55.148.54 65.119.214.9 65.192.195.0 65.192.195.1 65.192.195.10 65.192.195.11 65.192.195.12 65.192.195.13 65.192.195.14 65.192.195.15 65.192.195.16 65.192.195.17 65.192.195.18 65.192.195.19 65.192.195.2 65.192.195.20 65.192.195.21 65.192.195.22 65.192.195.23 65.192.195.24 65.192.195.25 65.192.195.26 65.192.195.27 65.192.195.28 65.192.195.29 65.192.195.3 65.192.195.30 65.192.195.31 65.192.195.4 65.192.195.5 65.192.195.6 65.192.195.7 65.192.195.8 65.192.195.9 65.214.32 65.214.36 65.214.38 65.214.44 65.214.45.0 65.214.45.1 65.214.45.10 65.214.45.100 65.214.45.11 65.214.45.12 65.214.45.13 65.214.45.14 65.214.45.15 65.214.45.16 65.214.45.17 65.214.45.18 65.214.45.19 65.214.45.2 65.214.45.20 65.214.45.21 65.214.45.22 65.214.45.23 65.214.45.24 65.214.45.25 65.214.45.26 65.214.45.27 65.214.45.28 65.214.45.29 65.214.45.3 65.214.45.30 65.214.45.31 65.214.45.32 65.214.45.33 65.214.45.34 65.214.45.35 65.214.45.36 65.214.45.37 65.214.45.38 65.214.45.39 65.214.45.4 65.214.45.40 65.214.45.41 65.214.45.42 65.214.45.43 65.214.45.44 65.214.45.45 65.214.45.46 65.214.45.47 65.214.45.48 65.214.45.49 65.214.45.5 65.214.45.50 65.214.45.51 65.214.45.6 65.214.45.7 65.214.45.8 65.214.45.9 # 100hot 206.253.217.18 # 123metaspider.com # UA "123metaspider-Bot (Version: 1.04, powered by www.123metaspider.com)" 82.165.35.184 # 2kcity.com 63.222.37.101 63.222.37.105 # A-Online 195.3.97.3 # Abacho 193.110.40.93 194.231.30.86 194.231.30.87 194.231.30.88 194.231.30.89 194.231.30.90 194.231.30.91 194.231.30.92 194.231.30.93 194.231.30.108 194.231.30.109 194.231.68.166 213.203.200.31 # Abadoor 209.68.2.36 # Abrexa.co.uk 194.117.133.180 # Acoon 194.231.42.185 # ActiveTourist # UA "Mozilla/4.0 (ActiveTouristBot;http://www.activetourist.com)" 86.131.209.38 86.131.210.252 # Activtourist # UA "Mozilla/4.0 (JemmaTheTourist;http://www.activtourist.com robot_txt_error@activtourist.com)" # UA "Mozilla/4.0 (JemmaTheTourist;http://www.activtourist.com)" 81.155.227.55 81.155.227.56 217.43.59.146 # Agada # UA "Mozilla/4.0 (agadine3.0) www.agada.de" 81.209.140.139 # Ah-Ha 208.204.161.8 216.250.141.186 # Aeiwi # UA "Marvin v0.3" 207.126.112.254 209.120.206.178 212.91.134.190 216.178.189.230 130.225.20.4 64.247.218.197 # Ahoy! The Homepage Finder 140.142.13.196 # Aladin 194.231.30.14 194.231.0.71 194.231.30.11 # Allesklar 195.37.68.3 195.37.68.65 195.37.68.180 195.37.68.181 # AllFreak.com 209.239.48.69 # Amfibi # UA "Amfibibot/0.06 (Amfibi Web Search; http://www.amfibi.com; agent@amfibi.com)" # UA "Amfibibot/0.07 (Amfibi Robot; http://www.amfibi.com; agent@amfibi.com)" 213.229.153.170 217.113.244.118 217.113.244.119 # ANAGRAROBOT 194.231.42.178 # Ananzi 167.160.195.61 # AnnoMille 80.16.145.187 # Ano-Kato.com (Greek) # UA "ano-kato.com v2.14 spider" 64.246.36.123 # AnswerBus # UA "AnswerBus (http://www.answerbus.com/)" 134.96.1.195 134.96.7.93 134.96.68.36 134.96.104.5 134.96.104.23 134.96.104.83 134.96.104.109 134.96.104.225 134.96.104.226 134.96.104.227 # AntiSearch # 62.210.155.50 # 62.210.155.56 # 62.210.155.58 # 62.210.155.59 # Apexoo.com # UA "Apexoo Spider 1.0" 216.240.143.30 # Apidom (Guadeloupe) 192.41.9.30 192.41.9.30 208.237.120.82 # Aport 194.67.18.84 194.67.18.70 194.67.18.65 194.67.18.69 194.67.18.68 194.67.18.67 194.67.18.66 194.67.18.63 194.67.18.71 194.67.18.72 194.67.18.73 194.67.18.74 # Arianna 195.210.89.11 195.210.91.24 195.210.91.80 195.210.91.112 195.210.91.113 195.210.91.187 195.210.91.212 195.210.91.213 195.210.91.214 195.210.91.235 195.210.91.215 192.92.126.4 192.92.126.5 # AtLocal # UA "AtlocalBot/1.1 +(http://www.atlocal.com/local-web-site-owner.html)" 216.55.159.187 216.55.159.189 216.55.159.190 # Axada # UA "axadine/ (Axadine Crawler; http://www.axada.de/; )" # 62.141.54.115 # Ay-Up # UA "Fred/0.01-dev (Fred; http://www.ay-up.com; fred@ay-up.com)" # UA "Fred/0.05 (Fred; http://www.ay-up.com; fred@ay-up.com)" 65.254.63.154 69.57.157.54 # Backrub 171.64.68.80 171.64.75.100 171.64.75.104 # Baidu # UA "BaiDuSpider" # UA "Baiduspider+(+http://www.baidu.com/search/spider.htm)" 122.152.128.19 122.152.128.48 162.105.207.185 162.105.207.192 202.103.134.196 202.108.11.106 202.108.11.108 202.108.11.132 202.108.22.70 202.108.22.73 202.108.23.74 202.108.22.75 202.108.22.79 202.108.22.80 202.108.22.81 202.108.23.71 202.108.249.176 202.108.249.177 202.108.249.178 202.108.249.179 202.108.249.182 202.108.249.183 202.108.249.184 202.108.249.185 202.108.249.186 202.108.249.187 202.108.249.188 202.108.249.189 202.108.249.190 202.108.250.195 202.108.250.196 202.108.250.197 202.108.250.198 202.108.250.199 202.108.250.206 202.108.250.207 202.108.250.226 202.108.250.241 202.108.250.242 202.108.250.243 202.108.250.244 202.108.250.253 211.100.24.90 211.100.24.91 211.100.24.92 211.100.24.93 211.100.24.94 211.100.24.95 211.100.24.96 211.100.24.97 211.100.25.196 211.100.25.197 211.100.25.198 211.100.25.199 211.100.25.200 211.100.25.202 211.100.25.204 211.101.48.53 60.28.17.36 60.28.17.39 60.28.17.43 60.28.17.41 60.28.17.45 60.28.17.47 61.135.145.204 61.135.146.205 61.135.146.206 61.135.145.207 61.135.145.208 61.135.145.209 61.135.145.211 61.135.145.212 61.135.145.215 61.135.145.216 61.135.145.221 61.135.145.251 61.135.146.197 61.135.146.199 61.135.146.200 61.135.146.201 61.135.146.202 61.135.146.205 61.135.146.210 61.135.145.219 61.135.146.251 61.135.162.78 61.149.2.221 # Batsch 204.251.93.10 # Become.com # UA "Mozilla/5.0 (compatible; BecomeBot/2.0.3; +http://www.become.com/webmasters.html)" # UA "Mozilla/5.0 (compatible; becomebot/2.0beta; +http://www.become.com/webmasters.html)" # UA "Mozilla/5.0 (compatible; becomebot/1.23; +http://www.become.com/webmasters.html)" # UA "Mozilla/4.7 [en](BecomeBot'at'exava.com)" # UA "Mozilla/5.0 (compatible; BecomeBot/2.2.5; MSIE 6.0 compatible; +http://www.become.com/site_owners.html)" 64.124.85 69.111.170.194 # Bellnet bellnet.com 208.28.152.71 # BigClique # UA "BigCliqueBOT/1.03-dev (bigclicbot; http://www.bigclique.com; bot@bigclique.com)" 72.3.246.162 216.130.179.196 # Bigsearch.ca # UA "Bigsearch.ca/Nutch-0.9-dev (Bigsearch.ca Internet Spider; http://www.bigsearch.ca/; info@enhancededge.com)" 72.0.207.162 216.58.87.217 # Bilgi.com # UA "BilgiBot/1.0(beta) (http://www.bilgi.com/; bilgi at bilgi dot com)" # 212.174.130.116 # 212.174.130.123 # 212.174.130.99 # BizSearch 194.198.230.36 # BlackSpider 210.159.73.36 210.159.73.35 # BlackWidow 140.190.65. # Blitzsuche blitzsuche.de 194.231.30.106 194.231.30.107 194.231.30.108 # Bloglines # UA "Bloglines Title Fetch/1.0 (http://www.bloglines.com)" 216.148.212.182 # Boitho # UA "boitho.com-robot/1.1" # UA "boitho.com-dc/0.4 ( http://www.boitho.com/dcbot.html )" # UA "boitho.com-dc/0.5 ( http://www.boitho.com/dcbot.html )" # UA "boitho.com-dc/0.52 ( http://www.boitho.com/dcbot.html )" # UA "boitho.com-dc/0.57 ( http://www.boitho.com/dcbot.html )" # UA "boitho.com-dc/0.58 ( http://www.boitho.com/dcbot.html )" # UA "boitho.com-dc/0.60 ( http://www.boitho.com/dcbot.html )" # UA "boitho.com-dc/0.66 ( http://www.boitho.com/dcbot.html )" # UA "boitho.com-dc/0.71 ( http://www.boitho.com/dcbot.html )" # UA "boitho.com-dc/0.72 ( http://www.boitho.com/dcbot.html )" # UA "boitho.com-dc/0.75 ( http://www.boitho.com/dcbot.html )" # UA "boitho.com-dc/0.81 ( http://www.boitho.com/dcbot.html )" # UA "boitho.com-dc/0.82 ( http://www.boitho.com/dcbot.html )" # UA "boitho.com-dc/0.83 ( http://www.boitho.com/dcbot.html )" 62.97.196.2 62.101.246.77 80.202.58.101 80.202.209.74 80.202.219.29 80.202.221.109 80.203.26.148 80.203.32.41 80.203.51.157 80.203.138.137 80.203.232.107 80.239.62.203 81.0.183.106 81.191.68.135 81.191.108.59 81.191.110.86 81.0.183.107 83.108.128.168 83.108.143.229 84.48.35.246 84.48.78.80 129.241.104.168 129.241.104.173 129.241.104.174 129.241.104.175 129.241.104.179 129.241.104.180 129.241.104.182 129.241.104.183 129.241.104.184 129.241.104.186 129.241.104.187 129.241.104.188 129.241.104.189 129.241.104.238 129.241.104.242 213.179.58.54 217.118.34.110 217.118.38.226 217.118.38.227 217.118.38.228 217.118.38.229 217.118.38.230 217.118.38.231 217.118.38.232 217.118.38.233 217.118.38.234 24.97.214.158 # Brainbot (Gigabaz) 151.189.96.99 134.93.7.97 134.93.7.98 # Brainysearch 64.57.64.90 # Brazilian Research Network 200.215.16.125 # BTbot # UA "BTbot/0.3 (+http://www.btbot.com/btbot.html)" # UA "btbot/0.4 ( http://www.btbot.com/btbot.html)" 84.128.33.71 84.150.79.18 84.150.122.26 217.235.240.139 217.235.241.160 # Burf.com # UA "Norbert the Spider(Burf.com)" 81.86.128.112 88.107.17.230 # BurstFind # UA "BurstFind Crawler 1.0/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; crawler@burstfind.com)" # UA "BurstFindCrawler/1.1 (crawler.burstfind.com; http://crawler.burstfind.com; crawler@burstfind.com)" 64.34.172.78 64.34.176.199 64.34.179.137 # CACTVS Chemistry Spider 131.188.128.229 194.199.4.2 # CanadianContent.net # UA "RoboCrawl (www.canadiancontent.net)" 207.44.220.53 # centrum.cz # UA "holmes/3.8 (morfeo.centrum.cz/bot)" 213.29.7.220 213.29.7.217 62.84.131.137 62.84.131.161 62.84.131.158 62.168.111.19 62.168.111.21 # Chimera 192.115.137.202 192.115.136.15 64.28.94.71 # Cipinet # UA "CipinetBot (http://www.cipinet.com/bot.html)" # 61.16.132.197 # Claymont 198.138.63.1 198.138.63.22 198.139.155.10 # Clush # UA "Mozilla/5.0 (Clustered-Search-Bot/1.0; support@clush.com; http://www.clush.com/)" # UA "Clushbot/2.1 (+http://www.clush.com/bot.html)" # UA "Clushbot/3.62-Laomedon (+http://www.clush.com/bot.html)" 209.249.80.231 209.249.80.232 209.249.80.233 209.249.80.245 209.249.80.246 # Colorstamps 209.10.61.5 # Combine System 130.235.86.152 # Comunet 195.76.40.251 # ConceptBot 66.47.255.121 # Convera # UA "CrawlConvera0.1 CrawlConvera@yahoo.com" # UA "ConveraCrawler/0.7 (+http://www.authoritativeweb.com/crawl)" # UA "ConveraCrawler/0.9d (+http://www.authoritativeweb.com/crawl)" 63.241.61.39 63.241.61.43 63.241.61.46 63.241.61.7 63.241.61.8 216.143.233.35 # Crawler.de # UA "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MSIECrawler)" 80.182.226.190 195.20.224.72 195.20.224.73 # Crawlzilla # UA "Crawlzilla/1.0 (Crawlzilla; http://www.crawlzilla.com; crawler@crawlzilla.com)" 66.235.180.244 # CS Hkust Wise 143.89.40.159 # CurryGuide 192.41.34.245 192.41.43.146 192.41.49.235 # Cydral # UA "CydralSpider/1.8 (Cydral Web Image Search; http://www.cydral.com)" 82.235.118.155 213.246.36.238 # DailyOrbit # UA "Orbiter/T-2.0 (+http://www.dailyorbit.com/bot.htm)" # UA "Orbiter (+http://www.dailyorbit.com/bot.htm)" 68.232.134.231 69.168.43.89 # DavesEngine 208.185.131.219 208.185.131.220 208.185.131.221 208.185.131.222 216.200.195.53 216.200.195.58 216.200.195.60 216.200.195.57 216.200.195.59 216.200.195.61 # DeepIndex # UA "DeepIndex" # UA "DeepIndex (www.deepindex.com)" 62.212.117.198 213.41.128.47 213.251.136.94 # DIE-KRAEHE # UA "-DIE-KRAEHE- META-SEARCH-ENGINE/1.0 http://www.die-kraehe.com" 217.160.94.14 # DivX Search Engine # UA "FlickBot 2.0 RPT-HTTPClient/0.3-3" 66.77.127.85 66.77.127.102 # Domanova 217.146.97.27 # DragonNext 210.199.215.54 # Drecom Career # UA "Drecombot/1.0 (http://career.drecom.jp/bot.html)" 210.233.67.132 # Dumbfind # UA "Dumbot(version 0.1 beta - dumbfind.com)" 38.118.73.250 68.167.196.88 68.239.122.138 151.200.115.249 # eDintorni # UA "fscrawler/3.0 (www.e-dintorni.it;Windows NT 5.0)" 213.215.160.170 # EDSA Phillipines 210.16.10.40 # eFamily 208.186.202.21 # Egoto.com Spain # UA "EgotoBot/4.8 (+http://www.egoto.com/about.htm)" 62.57.74.14 # Elyion # UA "NextGenSearchBot 1 (for information visit http://www.eliyon.com/NextGenSearchBot)" 207.31.251.140 # The Engine Rooms # UA "EngineBot/3.0" # UA "arachna_enginebot/v3.0[http://www.theenginerooms.com]" 81.153.61.72 81.153.62.103 # entireweb.com # UA "Speedy_Spider (http://www.entireweb.com)" # UA "Speedy Spider (Beta/1.0; speedy@entireweb.com)" # UA "Speedy Spider (Entireweb; Beta/1.0; http://www.entireweb.com/about/search_tech/speedyspider/)" 193.15.210.29 193.15.210.29 62.13.25.201 62.13.25.209 62.13.25.219 62.13.25.220 62.13.25.221 62.13.25.222 89.150.197.192 89.150.197.193 89.150.197.194 # Envolk.com # UA "envolk[ITS]spider/1.6(+http://www.envolk.com/envolkspider.html)" # UA "envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.php)" 70.169.191.4 71.102.140.247 # e-Society Project # UA "e-SocietyRobot(http://www.yama.info.waseda.ac.jp/~yamana/es/)" 42.142.128.210 133.9.238.71 136.187.19.99 210.128.142.42 # Esoterica 209.198.242.61 # Eule 194.231.76.158 # EuroFerret (MuscatFerret) 194.168.54.6 194.168.54.11 # EuroOne 195.95.124.3 195.95.124.7 # EuroSeek 212.209.54.40 212.209.54.134 195.242.46.55 # Evaal # UA "Evaal/0.7.1 (Evaal; http://search.evaal.com/bot.html; bot@evaal.com)" 66.249.26.98 # ExactSeek.com # UA "eseek-larbin_2.6.2 crawler@exactseek.com" # UA "exactseek-pagereaper-2.63 crawler@exactseek.com" # UA "ExactSeek_Spider" # UA "Mozilla/4.0" # UA "exactseek.com" 206.51.232.142 207.36.47.237 216.110.167.157 69.9.181.169 69.9.181.170 69.9.181.171 # Exalead # UA "Exalead NG/MimeLive Client (convert/http/0.120)" # UA "NG/2.0" # UA "Exabot NG/MimeLive Client (convert/http/0.170)" # UA "Mozilla/5.0 (compatible; Konqueror/3.4; Linux) KHTML/3.4.1 (like Gecko)" # UA "Exabot/2.0" # UA "Exabot-Test/1.0" 64.81.121.37 69.44.159.54 84.233.148.5 84.233.148.7 84.233.148.10 84.233.148.11 84.233.148.12 84.233.148.13 84.233.148.14 84.233.148.15 84.233.148.16 84.233.148.17 84.233.148.18 84.233.148.19 84.233.148.20 84.233.148.21 84.233.148.22 84.233.148.23 84.233.148.24 193.47.80.101 193.47.80.135 193.47.80.136 193.47.80.140 193.47.80.142 193.47.80.36 193.47.80.37 193.47.80.38 193.47.80.39 193.47.80.40 193.47.80.41 193.47.80.42 193.47.80.43 193.47.80.44 193.47.80.46 193.47.80.75 193.47.80.77 193.47.80.78 193.47.80.83 193.47.80.94 195.154.174.164 195.154.174.167 212.234.111.157 213.174.84.195 # Excite 198.3.103 199.172.148.105 199.172.148.11 199.172.148.37 199.172.149 199.172.152.238 199.172.152.34 199.172.152.54 199.172.152.56 199.172.152.57 199.172.152.95 199.172.153.174 199.172.153.178 199.172.156.168 199.172.156.169 199.172.156.170 199.172.156.219 199.172.157.28 204.62.245 # Exodus.Net 209.67.252.214 # EZResult 64.14.66.11 64.14.66.100 # FAST # UA "FastSearch Web Crawler for Verizon SuperPages (kevin.watters@fastsearch.com)" # UA "FAST Data Search Document Retriever/4.0 04072100 2004-07-21 07:54:10 UTC" # UA "FAST MetaWeb Crawler (helpdesk at fastsearch dot com)" 146.101.142.222 146.101.142.223 146.101.142.224 146.101.142.225 146.101.142.226 146.101.142.227 146.101.142.228 146.101.142.229 146.101.142.230 146.101.142.231 146.101.142.232 146.101.142.233 146.101.142.234 146.101.142.235 146.101.142.236 146.101.142.237 146.101.142.238 146.101.142.239 146.101.142.240 146.101.142.241 146.101.142.242 146.101.142.243 146.101.142.244 146.101.142.245 146.101.142.246 146.101.142.247 146.101.142.248 146.101.142.249 146.101.142.250 146.101.142.251 146.101.142.252 146.101.142.253 151.138.18.30 151.138.18.35 194.224.199.46 194.224.199.48 200.221.10.240 207.201.123.246 208.186.202.28 209.202.148 209.67.247 212.48.8.142 212.48.11.189 213.188.8.2 213.188.8.4 213.188.8.76 216.35.112.141 216.35.112.142 216.35.112.143 216.35.112.50 216.35.112.51 62.41.154 64.41.254.23 64.41.254.24 64.41.254.25 66.77.73 66.77.74 # Faxo # UA "Faxobot/1.0" 68.94.95.150 69.148.183.12 69.150.87.183 69.152.88.32 69.154.219.28 69.155.4.253 # FetchRover 192.41.15.30 # Filangy # UA "Filangy/1.01 (Filangy; http://www.filangy.com/filangyinfo.jsp?inc=robots.jsp; filangy-agent@filangy.com)" 64.29.182.10 # FindLink.com 207.86.79.6 207.86.79.12 # findquick.com 207.153.57.51 # Fireball # UA "Firefly/1.0 (compatible; Mozilla 4.0; MSIE 5.5)" 193.7.255.21 193.7.255.33 193.7.255.90 193.7.255.121 193.7.255.122 193.7.255.130 193.7.255.142 193.7.255.22 193.7.255.241 193.7.255.242 193.7.255.244 # First-Search 217.75.104.26 63.121.41.166 63.121.41.172 63.121.41.173 63.121.41.174 63.121.41.176 63.121.41.177 208.131.25.166 208.131.25.167 208.131.25.169 198.173.35.169 198.173.35.170 198.173.35.171 63.121.41.175 # Fish search 131.155.70.189 # Flix 195.190.21.70 # Foelock.co.uk # UA "Foelock.co.uk Search MyGo R2-H6-3" # UA "Foelock.co.uk Search MyGo Uknow 7" 84.13.2.175 84.13.26.47 84.13.56.131 89.240.138.167 # Francoroute 132.218.30.216 # FreeFind # UA "Mozilla/4.0 (compatible; MSIE 4.01; Windows 95; http://www.freefind.com/spider.html )" # UA "FreeFind/1.0 (spider@freefind.com)" 63.203.65.217 205.179.93.2 205.179.93.66 # Funkycat Search 151.196.224.70 209.176.27.63 64.208.104.2 # FunnelWeb 203.21.15.10 # Fybersearch # UA "FyberSpider (+http://www.fybersearch.com/fyberspider.php)" # 24.17.40.70 # 24.17.41.36 # 24.19.147.118 # Galaxy # UA "Mozilla/4.0 (compatible; MSIE 5.0; www.galaxy.com; www.ads-work.com; +http://www.galaxy.com/info/crawler.html)" 208.131.25.6 216.206.178.81 216.206.178.83 216.206.178.85 208.131.25.168 63.121.41.178 63.121.41.179 # gasta 194.46.8.25 # Genieknows.com # UA "Mozilla/5.0 wgao@genieknows.com" # UA "genieBot (http://64.5.245.11/faq/faq.html)" # UA "genieBot wgao@genieknows.com" 64.5.245.10 64.5.245.11 64.5.245.23 64.5.245.24 64.5.245.25 64.5.245.26 64.5.245.27 64.5.245.28 64.5.245.29 64.5.245.50 # Geobot # UA "Geobot/0.1 http://www.geobot.co.uk" # UA "GeoBot/version 1.0" 216.127.80.23 65.254.33.130 # Geona # UA "GeonaBot 1.0; http://www.geona.com/" # 69.59.142.172 # 216.93.161.126 # Gigablast # UA "Gigabot/1.0" # UA "Gigabot/2.0" # UA "GigabotSiteSearch/2.0 (sitesearch.gigablast.com)" # UA "Gigabot/2.0 (gigablast.com)" # UA "Gigabot/2.0/gigablast.com/spider.html" # 64.62.142.231 # 64.62.142.233 # 64.62.142.235 # 64.62.142.237 # 64.62.142.239 # 64.62.142.241 # 64.62.142.243 # 64.62.142.245 # 64.62.168.0 # 64.62.168.1 # 64.62.168.2 # 64.62.168.3 # 64.62.168.4 # 64.62.168.5 # 64.62.168.6 # 64.62.168.7 # 64.62.168.8 # 64.62.168.9 # 64.62.168.10 # 64.62.168.11 # 64.62.168.12 # 64.62.168.13 # 64.62.168.14 # 64.62.168.15 # 64.62.168.16 # 64.62.168.17 # 64.62.168.18 # 64.62.168.19 # 64.62.168.20 # 64.62.168.21 # 64.62.168.22 # 64.62.168.23 # 64.62.168.24 # 64.62.168.25 # 64.62.168.26 # 64.62.168.27 # 64.62.168.28 # 64.62.168.29 # 64.62.168.30 # 64.62.168.31 # 64.62.168.32 # 64.62.168.33 # 64.62.168.34 # 64.62.168.35 # 64.62.168.36 # 64.62.168.37 # 64.62.168.38 # 64.62.168.39 # 64.62.168.40 # 64.62.168.41 # 64.62.168.42 # 64.62.168.43 # 64.62.168.44 # 64.62.168.45 # 64.62.168.46 # 64.62.168.47 # 64.62.168.48 # 64.62.168.49 # 64.62.168.50 # 64.62.168.51 # 64.62.168.52 # 64.62.168.53 # 64.62.168.54 # 64.62.168.55 # 64.62.168.56 # 64.62.168.57 # 64.62.168.58 # 64.62.168.59 # 64.62.168.60 # 64.62.168.61 # 64.62.168.62 # 64.62.168.63 # 64.62.168.64 # 64.62.168.65 # 64.62.168.66 # 64.62.168.67 # 64.62.168.68 # 64.62.168.69 # 64.62.168.70 # 64.62.168.71 # 64.62.168.72 # 64.62.168.73 # 64.62.168.74 # 64.62.168.75 # 64.62.168.76 # 64.62.168.77 # 64.62.168.78 # 64.62.168.79 # 64.62.168.80 # 64.62.168.81 # 64.62.168.82 # 64.62.168.83 # 64.62.168.84 # 64.62.168.85 # 64.62.168.86 # 64.62.168.87 # 64.62.168.88 # 64.62.168.89 # 64.62.168.90 # 64.62.168.91 # 64.62.168.92 # 64.62.168.93 # 64.62.168.94 # 64.62.168.95 # 64.62.168.96 # 64.62.168.97 # 64.62.168.98 # 64.62.168.99 # 64.62.168.100 # 64.62.168.101 # 64.62.168.102 # 64.62.168.103 # 64.62.168.104 # 64.62.168.105 # 64.62.168.106 # 64.62.168.107 # 64.62.168.108 # 64.62.168.109 # 64.62.168.110 # 64.62.168.111 # 64.62.168.112 # 64.62.168.113 # 64.62.168.114 # 64.62.168.115 # 64.62.168.116 # 64.62.168.117 # 64.62.168.118 # 64.62.168.119 # 64.62.168.120 # 64.62.168.121 # 64.62.168.122 # 64.62.168.123 # 64.62.168.124 # 64.62.168.125 # 64.62.168.126 # 64.62.168.127 # 64.139.94.202 # 64.211.63.145 # 64.211.63.146 # 66.154.102 # 66.154.103 # 66.231.188 # 66.231.189 # 207.114.174.1 # 207.114.174.2 # 207.114.174.3 # 207.114.174.6 # 207.114.174.5 # 207.114.174.7 # 207.114.174.8 # 207.114.174.9 # 207.114.174.10 # 207.114.174.11 # 207.114.174.12 # 207.114.174.13 # 207.114.174.14 # 207.114.174.15 # 207.114.174.16 # 207.114.174.17 # 207.114.174.18 # 207.114.174.19 # 207.114.174.20 # 207.114.174.21 # 207.114.174.22 # 207.114.174.23 # 207.114.174.24 # 207.114.174.25 # 207.114.174.26 # 208.254.87.133 # 216.243.113.1 # 216.243.113.81 # 216.243.113.82 # 216.243.113.83 # 216.243.113.84 # 216.243.113.86 # 216.243.113.87 # 217.75.103.130 # Globalspec # UA "Ocelli/1.1 (http://www.globalspec.com)" # UA "Ocelli/1.3 (http://www.globalspec.com/Ocelli)" 66.194.55.242 66.195.77.130 # GoForIt.com # UA "GoForIt.com" # UA "GOFORITBOT ( http://www.goforit.com/about/ )" # 63.240.232.3 # 198.65.147.172 # 216.69.177.55 # Golden Can # UA "GoldenCanBot (http://www.GoldenCan.com/botinfo.aspx)" 64.224.197.99 # Goo/Infobee # UA "ichiro/1.0 (ichiro'at'nttr.co.jp)" # UA "ichiro/2.0 (ichiro@nttr.co.jp)" # UA "gazz/5.0 (gazz@nttr.co.jp)" # UA "ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html)" 202.229.31.13 202.229.31.14 202.229.31.15 210.150.10.100 210.150.10.73 210.150.25.156 210.165.39.216 210.165.39.217 210.165.39.216 210.165.39.244 210.165.39.245 210.165.39.246 210.165.39.247 210.165.39.248 210.165.39.249 210.165.39.250 210.165.39.252 210.165.39.253 210.165.39.254 210.173.179.22 210.173.179.26 210.173.179.30 210.173.179.31 210.173.179.32 210.173.179.33 210.173.179.37 210.173.179.40 210.173.179.45 210.173.179.47 210.173.179.50 210.173.179.53 210.173.179.82 210.173.179.83 210.173.179.149 210.173.180.157 210.173.180.160 210.173.180.168 210.173.180.192 # GoZero Search 216.181.122.11 216.181.122.110 # GoZilla 212.125.68.30 # Grub 206.30.142.100 # Harvest 193.62.81.96 193.62.81.96 # HiddenStreet.com 64.246.15.61 # HI (HTML Index) Search 192.41.170.17 # Hitbox 208.232.223.80 # Hitsearch 216.55.4.15 # Homerweb # UA "Homerbot: www.homerweb.com" 24.1.248.148 # Honda Search # UA "Honda-Search/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; search@honda-search.com)" 69.16.227.165 # HotBot 209.1.12.2 209.1.12.220 209.1.12.221 209.1.12.222 209.1.12.224 209.1.12.225 209.1.12.226 209.1.12.227 209.1.12.228 209.1.12.229 209.1.12.230 209.1.12.231 209.1.12.232 209.1.12.233 209.1.12.234 209.1.12.235 209.1.12.236 209.1.12.237 209.1.12.238 209.1.12.239 209.1.12.240 209.1.12.241 209.1.12.242 # HotWired 204.62.132.20 # HTDig 209.75.193.22 # Hubat 209.114.176.250 # i1search.com # UA "i1searchbot/2.0 (i1search web crawler; http://www.i1search.com; crawler@i1search.com)" 65.111.164.143 65.111.168.41 # IBM_Planetwide 204.146.80.99 204.146.81.99 # icorus.com 209.239.36.253 # ICQ 205.188.147.53 205.188.147.56 205.188.252.121 # Ideare 212.123.67.70 # IErachnid # UA "IErachnid 3.0 http://uk.searchengineeurope.com" 193.120.253.2 # igougo.com # UA "Nutch/0.7 (igougocrawler; http://www.igougo.com/search/web.asp; nutch-agent@lucene.apache.org)" 209.208.197.115 # ILSE # UA "Mozilla/3.0 (INGRID/3.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmelden.ilse.nl/?aanmeld_mode=webhints)" 62.69.162.144 62.69.162.171 62.69.162.177 213.244.181.190 212.72.39.2 194.134.109.3 194.134.109.7 194.134.109.17 194.134.109.21 212.72.39.8 212.72.39.3 212.72.39.4 212.72.39.5 212.72.39.6 212.72.39.7 212.72.39.9 212.72.39.10 212.72.39.11 212.72.39.12 212.72.39.13 212.72.39.14 212.72.39.15 212.72.39.16 212.72.39.17 212.72.39.18 212.72.39.19 212.72.39.20 212.72.39.21 212.72.39.22 212.72.39.23 212.72.39.24 212.72.39.27 212.72.39.28 212.72.39.29 212.72.39.30 212.72.39.194 212.72.39.195 212.72.39.196 212.72.39.197 212.72.39.198 212.72.39.199 212.72.39.200 212.72.39.201 212.72.39.202 212.72.39.203 212.72.39.204 212.72.39.205 212.72.39.206 212.72.39.207 212.72.39.208 212.72.39.209 212.72.39.210 212.72.39.211 212.72.39.212 212.72.39.213 212.72.39.214 212.72.39.215 212.72.39.216 212.72.39.217 212.72.39.218 212.72.39.219 212.72.39.220 212.72.39.221 # Iltrovatore.it # UA "iltrovatore-setaccio/1.2-dev (spidering; http://www.iltrovatore.it/aiuto/motore_di_ricerca.html; info@iltrovatore.it)" # UA "Iltrovatore-Setaccio/0.3-dev (Indexing; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)" # UA "IlTrovatore-Setaccio/1.2 (It-bot;compatible;MSIE 6.0;Mozilla/4.0; http://www.iltrovatore.it/bot.html; bot@iltrovatore.it)" # UA "IlTrovatore-Setaccio/1.2 (Italy search engine; http://www.iltrovatore.it/bot.html; bot@iltrovatore.it)" 213.215.201.204 213.215.201.205 213.215.201.221 213.215.201.222 213.215.201.234 213.215.201.235 # Inetseek.de # UA "InetSeekBot2005 (Version: 1.01, Powered by InetSeek.de)" # UA "Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/0.3" 195.27.115.51 212.72.181.12 # Infomak 165.254.215.60 165.254.215.61 165.254.215.62 165.254.215.63 # Informant 129.170.24.57 # Infoseek # UA "Infoseek SideWinder/2.0B (Linux 2.4 i686)" 195.145.119.24 195.145.119.25 198.5.208 198.5.210 202.33.250.146 202.33.250.147 202.33.250.148 202.33.250.149 202.33.250.150 202.33.250.151 202.33.250.152 202.33.250.153 202.33.250.154 204.162.96 204.162.97.1 204.162.97.152 204.162.97.17 204.162.97.2 204.162.97.205 204.162.97.228 204.162.97.231 204.162.97.3 204.162.97.32 204.162.98.11 204.162.98.12 204.162.98.124 204.162.98.126 204.162.98.151 204.162.98.161 204.162.98.168 204.162.98.18 204.162.98.192 204.162.98.2 204.162.98.237 204.162.98.27 204.162.98.3 204.162.98.36 204.162.98.38 204.162.98.4 204.162.98.45 204.162.98.48 204.162.98.49 204.162.98.5 204.162.98.6 204.162.98.7 204.162.98.8 204.162.98.80 204.162.98.88 204.162.98.9 204.162.98.91 204.162.98.98 204.202.132.19 205.226.201 205.226.203.186 205.226.203.35 205.226.203.56 205.226.203.62 205.226.204.238 206.3.30.196 206.3.30.250 206.3.30.251 210.148.160.157 210.148.160.163 210.148.160.165 210.148.160.206 210.155.157 210.155.159 210.236.233.130 210.236.233.131 210.236.233.132 210.236.233.133 210.236.233.135 210.236.233.136 210.236.233.137 210.236.233.139 210.236.233.150 210.236.233.151 210.236.233.155 210.236.233.160 210.236.233.161 211.13.222.230 211.18.214.194 212.185.44.10 212.185.44.11 212.185.44.12 212.185.44.15 # InfoSpiders 128.255.244.88 # Insitor # UA "Insitor.com, search and find world wide!" 80.67.17.98 # Inter-Fux 195.57.152.80 # Internet Archiver # 206.132.56.130 # Internet Portal 3000 209.133.111.132 216.200.196.8 216.200.196.9 216.200.196.10 216.200.196.11 216.200.196.12 216.200.196.13 216.200.196.14 216.200.196.15 # Intersearch 194.232.15.65 # Interseek.com # UA "Mozilla/5.0 (compatible; Najdi.si/3.1)" 85.10.36.125 # ISIZE 133.194.226.226 # iSparkle 193.195.19.10 # ISpi Crawler 207.82.104.100 # it LOOX 66.240.140.190 # iWon 216.33.59.61 # JAYDE.COM 208.239.240.102 66.28.139.66 66.28.139.71 # JAYDE.DE 195.226.160.249 # jblue 216.234.161.144 # Jeteye # UA "Jetbot/1.0" # 64.71.144.11 # 64.71.144.13 # 64.71.144.15 # 64.71.144.17 # 64.71.144.19 # 64.71.144.21 # 64.71.144.23 # 64.71.144.25 # 64.71.144.27 # 64.71.144.29 # 64.71.144.31 # 64.71.144.33 # 64.71.144.34 # 64.71.144.35 # 64.71.144.36 # 64.71.144.37 # 64.71.144.39 # 64.71.144.43 # 64.71.144.45 # 64.71.144.47 # 64.71.144.49 # 64.71.144.51 # 64.71.144.53 # 64.71.144.55 # 64.71.144.57 # 64.71.144.59 # 64.71.144.61 # 64.71.144.63 # 64.71.144.65 # 64.71.144.67 # 64.71.144.69 # 64.71.144.70 # 64.71.144.71 # 64.71.144.72 # 64.71.144.73 # 64.71.144.75 # 64.71.144.77 # Jippii.fi 195.211.11.50 # Job Robot (Jobot) 216.148.213.150 # The Jubii Indexing Robot 212.112.128.30 # JumpStation 139.153.13.200 # Jyxo.com # UA "Jyxobot/1" 195.113.214.196 195.113.214.204 195.113.214.207 195.122.204.2 195.122.208.217 212.47.13.201 212.71.128.66 212.71.128.67 # Kanoodle 216.33.119.136 216.33.119.138 38.170.72.194 # Kazoom # UA "KazoomBot/0.06-dev (Kazoom; http://www.kazoom.ca/bot.html; kazoombot@kazoom.ca)" 207.81.156.97 # Keywen Encyclopedia # UA "EasyDL/3.04 http://keywen.com/Encyclopedia/Bot" 24.57.2.49 24.57.5.63 24.57.8.78 24.57.240.53 24.235.212.163 # Kolibri 151.189.12.140 # kosmix.com # cosmix.com # UA "cfetch/1.0" # UA "voyager/1.0" 38.113.234.180 38.113.234.181 # Kuloko # UA "kulokobot www.kuloko.com kuloko@backweave.com" # UA "kuloko-bot/0.2" 66.90.81.41 66.227.104.196 # Linknz.co.nz # UA "Linknzbot (+ http://www.linknz.co.nz/robot.php)" 202.191.32.67 202.191.34.146 # Localbot # UA "LocalBot/1.0 (+http://www.localbot.co.uk/)" 62.49.200.56 80.177.149.217 # Local.com # UA "LocalcomBot/1.2 (+http://www.local.com/bot.htm)" # UA "LocalcomBot/1.2.2 (+http://www.local.com/bot.htm)" 216.52.252.195 216.52.252.208 63.251.210.65 # Look.com # UA "NetResearchServer(http://www.look.com)" # UA "LookBot" # UA "Look.com" 64.40.105.197 206.191.13.2 206.191.13.3 209.87.232.5 # Lookseek # UA "Seeker.lookseek.com" 12.199.64.41 12.199.64.69 209.177.62.135 209.177.62.151 # Looksmart # UA "Mozilla/4.0 (compatible; grub-client-1.2.1; Crawl your own stuff with http://grub.org)" # UA "NutchCVS/0.05 (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)" # UA "MARTINI" 207.138.42.212 64.241.242.160 64.241.242.18 64.241.242.34 # Lotse 195.20.224.66 # Magpie 195.153.48.120 # mande 216.94.9.119 # Mcmail 195.44.0.150 195.44.0.104 195.44.0.152 195.44.0.153 # MerzScope 207.158.203.71 # Metager # UA "Metager2 (http://metager2.de/site/webmaster.php)" 130.75.2.24 130.75.6.10 # Metaspinner # UA "Metaspinner/0.01 (Metaspinner; http://www.meta-spinner.de/; support@meta-spinner.de/)" 213.203.245.153 213.203.245.158 # Mirago # UA "HenryTheMiragoRobot" # UA "HenryTheMiragoRobot (http://www.miragorobot.com/scripts/mrinfo.asp)" # UA "Mirago-Test-Robot (http://www.miragorobot.com)" 194.202.39.44 212.56.39.2 217.154.244.217 217.154.244.228 217.154.244.229 217.154.244.230 217.154.244.231 217.154.244.232 217.154.244.233 217.154.244.234 217.154.244.235 217.154.244.239 217.154.244.240 217.154.244.241 217.154.245.2 217.154.245.201 217.154.245.202 217.154.245.203 217.154.245.204 217.154.245.205 217.154.245.207 217.154.245.209 217.154.245.210 217.154.245.211 217.154.245.212 217.154.245.213 217.154.245.214 217.154.245.215 217.154.245.216 217.154.245.217 217.154.245.218 217.154.245.219 217.154.245.220 217.154.245.224 217.154.245.225 217.154.245.226 217.154.245.227 217.154.245.228 217.154.245.229 217.154.245.230 217.154.245.232 217.154.245.233 217.154.245.234 217.154.245.236 217.154.245.237 217.154.245.238 217.154.245.239 217.154.245.240 217.154.245.241 217.154.245.242 217.154.245.244 217.154.245.245 217.154.245.246 217.154.245.248 217.154.245.249 217.154.244.251 217.154.245.250 217.154.245.251 217.205.60 217.205.61 # Mister-Jack 195.200.163.250 195.200.169.193 # MixCat 67.18.55.52 198.31.135.245 # Mojeek # UA "citenikbot/0.4 (albert; http://www.citenik.co.uk/bot.html)" # UA "MojeekBot/0.1 (galileo; http://www.mojeek.com/bot.html)" # UA "MojeekBot/0.2 (archi; http://www.mojeek.com/bot.html)" # UA "Mozilla/5.0 (compatible; MojeekBot/2.0; http://www.mojeek.com/bot.html)" 217.155.205.49 217.42.129.249 217.42.133.84 81.149.13.26 81.154.83.118 81.155.214.30 83.67.53.154 # MotionBridge 213.41.71.18 # Mozdex # UA "mozDex/0.04-dev (mozDex; http://www.mozdex.com/en/bot.html; spider@mozdex.com)" # "mozDex/0.05-dev (mozDex; http://www.mozdex.com/bot.html; spider@mozdex.com)" # 63.246.154.32 # 208.37.26.100 # 208.37.26.120 # 209.67.208.170 # myEfinder.com 24.42.211.66 # Nathan 194.221.102.137 # NationalDirectory 66.27.55.14 209.116.58.140 199.184.188.143 199.184.188.151 199.184.188.160 204.210.31.231 209.116.58.143 209.191.102.228 # Naver # UA "Cowbot-0.1 (NHN Corp. / 2-3011-1954 / nhnbot@naver.com)" # UA "NaverBot-1.0 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)" # UA "Mozilla/4.0 (compatible; NaverBot/1.0; nhnbot@naver.com)" # UA "Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/delete_main.asp)" # UA "Yeti/0.01 (nhn/1noon, yetibot@naver.com, check robots.txt daily and follows it)" 61.78.61.162 61.78.61.163 61.78.61.164 61.78.61.165 61.78.61.166 61.78.61.167 61.78.61.168 61.78.61.176 61.78.61.192 61.78.61.193 61.78.61.194 61.78.61.195 61.78.61.206 61.78.61.220 61.78.61.221 61.78.61.222 61.78.61.223 202.179.180.53 202.179.181.137 202.179.181.138 218.23.2.122 218.145.25.11 218.145.25.14 218.145.25.17 218.145.25.19 218.145.25.20 218.145.25.43 218.145.25.45 218.145.25.46 218.145.25.49 218.145.25.51 218.145.25.52 218.145.25.53 218.145.25.78 218.145.25.83 218.145.25.105 218.145.25.109 218.145.25.110 218.145.25.111 218.145.25.113 220.73.146.105 220.73.146.106 220.73.146.107 220.73.146.108 220.73.159.54 220.73.159.55 220.73.159.56 220.73.159.57 220.73.159.58 220.73.159.59 220.73.159.60 220.73.159.61 220.73.159.62 220.73.159.63 220.73.165.11 220.73.165.12 220.73.165.14 220.73.165.15 220.73.165.17 220.73.165.77 220.73.165.142 220.73.165.143 220.73.165.204 220.73.165.206 220.95.235.166 222.122.194.111 222.122.194.112 222.122.194.114 222.122.194.115 222.122.194.164 222.122.194.27 222.122.194.31 222.122.194.33 222.122.194.35 222.122.194.41 222.122.194.47 222.122.194.53 222.122.194.75 # NDV 194.231.30.16 # Netcraft 195.188.192.23 # NetMind 208.222.98.150 # NetNose # UA "Mozilla/4.0 (compatible; MSIE 5.0;NetNose-Crawler 2.0; A New Search Experience: http://www.netnose.com)" 199.35.98.30 199.35.98.241 # NetScoop 150.59.20.63 150.59.20.20 # Netseek 216.71.78.11 # NetWhat.com # UA "NetWhatCrawler/0.06-dev (NetWhatCrawler from NetWhat.com; http://www.netwhat.com; support@netwhat.com)" 69.9.167.198 # NG-Search.com # UA "NG-Search/0.90 (NG-SearchBot; http://www.ng-search.com; )" 84.56.103.71 84.56.114.19 84.56.74.242 84.56.77.39 84.56.84.58 84.56.93.33 # Nit-Net 212.63.155.1 # Nokodo # UA "NokodoBot/1.2 (+http://nokodo.com/bot.htm)" 67.18.222.18 # NorthernLight 195.228.240.177 204.166.111.29 205.181.75.130 205.181.75.60 205.181.75.66 205.181.75.75 205.181.75.76 208.219.77 216.34.102 216.34.109.190 216.34.109.191 216.34.109.192 64.95.79.195 64.95.79.40 # The NorthStar Robot 129.110.16.16 129.110.10.1 # Noxtrum # UA "noxtrumbot/1.0 (crawler@noxtrum.com)" 194.224.199.47 194.224.199.50 194.224.199.52 # NR1-Online # UA "Nr1-Onlinebot (Version: 1.02, powered by www.nr1-online.de)" # 62.112.150.218 # NTT Cyberspace Laboratories 210.165.39.210 210.165.39.211 210.165.39.212 210.165.39.213 210.165.39.214 210.165.39.215 210.165.39.253 # Nusearch # UA "NuSearch Spider www.nusearch.com" 82.32.121.70 82.32.121.164 82.32.123.249 82.33.193.111 82.68.206.22 84.9.137.104 208.53.131.158 # Nutch # UA "Nutch" # UA "NutchCVS/0.03-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.sourceforge.net)" # UA "NutchCVS/0.03-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)" # UA "NutchOrg/0.03-dev (Nutch; www.nutch.org/docs/bot.html; nutch-agent@lists.sourceforge.net)" # UA "NutchCVS/0.05-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)" # The NWI Robot 130.235.86.136 130.235.86.137 # Nzexplorer 202.36.240.1 # Oberland Net 194.221.132.133 # Objects Search # UA "ObjectsSearch/0.01 (ObjectsSearch; http://www.ObjectsSearch.com/bot.html; support@thesoftwareobjects.com)" # UA "ObjectsSearch/0.02 (ObjectsSearch; http://www.ObjectsSearch.com/bot.html; support@thesoftwareobjects.com)" # UA "ObjectsSearch/0.06 (ObjectsSearch; http://www.ObjectsSearch.com/bot.html; support@thesoftwareobjects.com)" 68.88.244.177 68.88.244.178 69.150.7.163 69.150.7.164 69.150.7.165 # Occam 128.95.1.184 128.95.1.208 128.95.1.84 # OmniExplorer # UA "OmniExplorer_Bot/1.07 (+http://www.omni-explorer.com) Internet Categorizer" # UA "OmniExplorer_Bot/1.09 (+http://www.omni-explorer.com) Internet Categorizer" # UA "OmniExplorer_Bot/1.09 (+http://www.omni-explorer.com)" # UA "OmniExplorer_Bot/1.10 (+http://www.omni-explorer.com) Jobs Crawler" # UA "OmniExplorer_Bot/1.18 (+http://www.omni-explorer.com) Torrent Crawler" # UA "OmniExplorer_Bot/2.3 (+http://www.omni-explorer.com) WorldIndexer" # UA "OmniExplorer_Bot/2.57 (+http://www.omni-explorer.com) WorldIndexer" # UA "OmniExplorer_Bot/2.67 (+http://www.omni-explorer.com) WorldIndexer" # UA "OmniExplorer_Bot/2.69 (+http://www.omni-explorer.com) WorldIndexer" # UA "OmniExplorer_Bot/2.70 (+http://www.omni-explorer.com) WorldIndexer" # UA "OmniExplorer_Bot/2.71 (+http://www.omni-explorer.com) WorldIndexer" # UA "OmniExplorer_Bot/2.73 (+http://www.omni-explorer.com) WorldIndexer" # UA "OmniExplorer_Bot/2.78a (+http://www.omni-explorer.com) WorldIndexer" # UA "OmniExplorer_Bot/2.82 (+http://www.omni-explorer.com) WorldIndexer" # UA "OmniExplorer_Bot/2.83 (+http://www.omni-explorer.com) WorldIndexer" # UA "OmniExplorer_Bot/2.86b (+http://www.omni-explorer.com) WorldIndexer" # UA "OmniExplorer_Bot/2.91 (+http://www.omni-explorer.com) WorldIndexer" # UA "OmniExplorer_Bot/2.92 (+http://www.omni-explorer.com) WorldIndexer" # UA "OmniExplorer_Bot/2.93 (+http://www.omni-explorer.com) WorldIndexer" # UA "OmniExplorer_Bot/3.06c (+http://www.omni-explorer.com) WorldIndexer" # UA "OmniExplorer_Bot/3.06d (+http://www.omni-explorer.com) WorldIndexer" # UA "OmniExplorer_Bot/4.32 (+http://www.omni-explorer.com) WorldIndexer" # UA "OmniExplorer_Bot/5.24 (+http://www.omni-explorer.com) WorldIndexer" # UA "OmniExplorer_Bot/6.46 (+http://www.omni-explorer.com) WorldIndexer" # UA "OmniExplorer_Bot/6.47 (+http://www.omni-explorer.com) WorldIndexer" # UA "OmniExplorer_Bot/6.57 (+http://www.omni-explorer.com) WorldIndexer" 64.62.175.130 64.62.175.131 64.62.175.137 64.71.131.109 64.71.131.117 64.127.124.130 64.127.124.131 64.127.124.132 64.127.124.133 64.127.124.139 64.127.124.143 64.127.124.145 64.127.124.148 64.127.124.153 64.127.124.157 64.127.124.158 64.127.124.159 64.127.124.165 64.127.124.168 64.127.124.169 64.127.124.170 64.127.124.171 64.127.124.172 64.127.124.173 64.127.124.188 64.127.124.189 64.127.124.190 64.127.124.191 64.127.124.193 65.19.150.134 65.19.150.204 65.19.150.206 65.19.150.207 65.19.150.208 65.19.150.209 65.19.150.210 65.19.150.211 65.19.150.212 65.19.150.213 65.19.150.214 65.19.150.217 65.19.150.218 65.19.150.219 65.19.150.220 65.19.150.221 65.19.150.222 65.19.150.223 65.19.150.224 65.19.150.225 65.19.150.226 65.19.150.227 65.19.150.228 65.19.150.229 65.19.150.230 65.19.150.231 65.19.150.232 65.19.150.233 65.19.150.234 65.19.150.235 65.19.150.236 65.19.150.237 65.19.150.238 65.19.150.239 65.19.150.240 65.19.150.241 65.19.150.242 65.19.150.243 65.19.150.244 65.19.150.245 65.19.150.246 65.19.150.247 65.19.150.248 65.19.150.249 65.19.150.250 65.19.150.251 65.19.150.252 65.19.169.228 65.19.169.229 65.19.169.230 65.19.150.250 65.19.169.242 65.19.169.252 65.19.169.254 # onet.pl # UA "Onet.pl SA, http://szukaj.onet.pl" # UA "Mozilla/5.0 (compatible; OnetSzukaj/5.0; +http://szukaj.onet.pl)" # UA "holmes/3.9 (OnetSzukaj/5.0; +http://szukaj.onet.pl)" 213.180.128.151 213.180.128.152 213.180.128.153 213.180.128.154 213.180.137.71 # onlinepilot.de 195.20.225.123 212.227.109.241 195.20.225.112 195.20.225.115 212.227.109.14 212.227.109.57 212.227.109.58 212.227.109.197 212.227.109.225 212.227.109.229 212.227.118.9 212.227.118.25 212.227.118.130 212.227.119.5 # Only 64.23.82.44 # Openfind # UA "Gaisbot/3.0+(robot@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)" # UA "Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html)" # UA "Openfind data gatherer,Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html)" 140.123.100.5 140.123.101.14 140.123.101.143 140.123.101.144 140.123.101.145 140.123.101.146 140.123.101.147 140.123.101.148 140.123.101.62 140.123.101.66 140.123.101.67 140.123.103.2 140.123.103.244 140.123.103.4 202.165.96.139 202.165.96.140 202.165.96.167 202.165.96.173 205.158.61.194 205.158.61.197 205.158.61.198 205.158.61.200 205.158.61.201 205.158.61.202 205.158.61.207 209.133.111.56 209.133.111.57 209.133.111.58 210.201.54.203 210.201.54.207 210.201.54.209 210.201.54.214 210.201.54.216 210.201.54.218 210.59.144.148 210.59.144.149 211.155.160.12 211.155.160.2 211.155.160.5 211.72.252.147 211.72.252.150 211.72.252.182 211.72.252.242 211.72.252.243 211.72.252.49 212.98.78.29 216.250.80.67 220.135.237.66 220.135.254.94 221.169.30.130 61.59.121.2 64.208.8.196 64.62.168.160 64.62.168.173 66.234.139.204 66.234.139.205 66.234.139.206 66.234.139.207 66.234.139.208 66.234.139.209 66.234.139.212 66.234.139.213 66.234.139.214 66.234.139.215 66.234.139.216 66.234.139.218 66.237.60 66.7.131.130 66.7.131.131 66.7.131.132 66.7.131.133 66.7.131.134 66.7.131.135 66.7.131.136 66.7.131.137 66.7.131.138 66.7.131.139 66.7.131.140 66.7.131.141 66.7.131.142 66.7.131.143 66.7.131.144 66.7.131.145 66.7.131.146 66.7.131.147 66.7.131.148 66.7.131.149 66.7.131.150 66.7.131.151 66.7.131.152 66.7.131.153 66.7.131.154 66.7.131.155 66.7.131.156 66.7.131.157 66.7.131.158 66.7.131.159 66.7.131.160 66.7.131.161 66.7.131.162 66.7.131.163 66.7.131.164 66.7.131.165 # Open Text 204.138.115.2 # Peerbot.com # UA "PEERbot www.peerbot.com" 213.239.197.150 213.239.206.109 # Physics.org # UA "BravoBrian SpiderEngine MarcoPolo" 213.215.133.19 # Picsearch # UA "psbot/0.1 (+http://www.picsearch.com/bot.html)" # UA "http://www.picsearch.com/bot.html" 212.69.208.31 213.242.179.43 217.75.104.23 217.75.104.26 217.212.224.142 217.212.224.143 217.212.224.144 217.212.224.145 217.212.224.146 217.212.224.147 217.212.224.148 217.212.224.149 217.212.224.159 217.212.224.164 217.212.224.165 62.119.21.136 62.119.21.132 62.119.21.135 62.119.21.137 62.119.21.138 62.119.21.139 62.119.21.150 62.119.21.157 62.119.133.11 62.119.133.12 62.119.133.13 62.119.133.14 63.223.65.253 83.140.161.141 83.140.161.142 # Pioneer 128.109.136.132 # PGP Key Agent 195.120.233.1 # Pinpoint 209.116.70.46 # Pipeline # UA "pipeLiner/0.10 (PipeLine Spider; http://www.pipeline-search.com/webmaster.html)" 24.106.39.250 # Pizilla 161.58.207.17 216.218.130.79 216.218.155.2 216.218.197.2 # PlanetClick 207.87.8.78 207.87.10.33 # Pompos # UA "Pompos/1.3 http://dir.com/pompos.html" 212.27.33.160 212.27.33.161 212.27.33.162 212.27.33.163 212.27.33.164 212.27.33.165 212.27.33.166 212.27.33.167 212.27.33.168 212.27.33.169 212.27.41.11 212.27.41.14 212.27.41.22 212.27.41.23 212.27.41.24 212.27.41.25 212.27.41.26 212.27.41.30 212.27.41.31 212.27.41.33 212.27.41.34 212.27.41.35 212.27.41.36 212.27.41.37 212.27.41.38 212.27.41.39 212.27.41.40 212.27.41.41 # Pop.gr # UA "Pop.gr/6.1 Arachne(compatible; MSIE 6.0; Windows 98; Corus)" 195.186.149.91 # PortalB 209.10.169.15 209.10.169.16 209.10.169.17 # Portal Juice 209.20.44.236 # Prasso # UA "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; PrassoSunner 1.00; http://www.prasso.com" 67.18.87.100 # Prolo # UA "ProloCrawler/1.0 (http://www.prolo.com)" 209.203.226.174 # Proodle # UA "www.proodle.com" 193.12.151.201 # psi.net 154.15.28.143 # Psychcrawler.com # UA "MSIE (MSIE 5.23; Mac_PowerPC)" 69.225.183.82 # Qango 212.78.64.35 217.158.17.25 # Qweery # UA "QweeryBot v2.05 -- http://qweerybot.qweery.com" # UA "QweeryBot/2.06 ( http://qweerybot.qweery.com)" 80.60.157.168 84.82.133.41 # Quepasa.com # UA "QuepasaCreep ( crawler@quepasacorp.com )" 69.28.130.222 69.28.130.229 69.28.130.230 69.28.130.231 # Questfinder 216.104.145.71 # Rambler # UA "StackRambler/2.0 (MSIE incompatible)" 81.19.66.9 81.19.66.6 81.19.66.39 81.176.67.106 81.19.66.8 81.19.67.34 81.19.66.42 81.19.66.38 81.222.64.10 81.19.66.74 81.19.67.253 81.19.67.247 # RawGrunt.com # UA "Blaiz-Bee/2.00.953 (+http://www.blaiz.net)" 203.87.123.139 # Realnames 63.251.238.8 216.86.229.85 216.86.229.86 # RedCarpet # UA "RedCarpet/1.2 (http://www.redcarpet-inc.com/robots.html)" 66.45.38.86 # RedKernel # UA "RedKernel WWW-Spider 2/0 (+http://www-spider.redkernel-softwares.com/)" 66.55.143.162 # Relevare # UA "Bumblebee@relevare.com" 80.229.145.226 80.229.216.40 83.146.31.19 212.158.198.8 212.158.235.113 212.158.249.0 212.158.251.24 # Rex 212.172.94.128 206.215.122.20 # Rufus # UA "RufusBot (Rufus Web Miner; http://64.124.122.252/feedback.html)" 64.124.122.228 # SafetyNet Robot 209.254.2.2 # Sagool.jp # UA "MaSagool/1.0 (MaSagool; http://sagool.jp/; masagool@sagool.jp)" 124.32.246.45 # Scrub the web # UA "Scrubby/2.2 (http://www.scrubtheweb.com/)" # UA "Scrubby/2.1 (http://www.scrubtheweb.com/abs/meta-check.html)" 64.140.165.132 64.140.165.133 64.140.165.139 66.93.156.38 66.93.156.39 208.234.1.83 208.145.190.250 208.145.190.251 208.145.190.254 # Search.ch # UA "search.ch V1.4.2 (spiderman@search.ch; http://www.search.ch)" 195.141.85.116 195.141.85.142 195.141.85.146 # searchengine.com 194.201.93.6 194.201.93.18 194.201.93.118 # SearchHippo 206.40.146.58 208.148.122.27 208.148.122.28 208.148.122.29 # SearchIt # UA "SearchIt.Bot 1.62 (http://www.searchit.com/)" # UA "SearchIt-Bot/2.3.1 (Searchit.Com; http://www.searchit.com/; )" 81.169.136.109 205.237.206.30 # SearchKing 208.165.96.26 # Searchmee # UA "Searchmee! Spider v0.98a" # 64.202.100.84 # SearchMil.com 206.129.98.7 206.129.98.16 206.129.98.19 # Searchopolis 206.129.0.3 206.129.0.131 206.129.0.132 206.129.1.24 206.129.1.27 # SearchSite # UA "SearchSight/2.0 (http://SearchSight.com/)" # 216.24.32.42 # SearchSpider # UA "Searchspider/1.2 (SearchSpider; http://www.searchspider.com; webmaster@searchspider.com)" 24.90.243.203 # SearchUK.com # UA "searchuk.comBot 1.1 (http://searchuk.com/spider)" # UA "searchuk.comBot 1.2 (http://searchuk.com/spider)" 81.92.97.41 84.73.59.129 # SearchWho 216.205.148.106 # SearchWire.co.uk # UA "WireBot/1.0 (SearchWire.co.uk)" 82.42.115.108 # Seekon.com # UA "SeekOn Spider 1.9(+http://www.seekon.com/spider.html)" 64.151.82.12 # Seekport # UA "Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/2.2" 195.27.115.50 195.27.215.70 195.27.215.89 195.27.215.91 195.27.215.92 # Semager.de # UA "Semager/1.0 (http://www.semager.de)" 212.114.209.250 # Senrigan 133.9.222.37 # Sensis.com.au # UA "Sensis Web Crawler (search_comments\at\sensis\dot\com\dot\au)" # UA "Sensis.com.au Web Crawler (search_comments\at\sensis\dot\com\dot\au)" 66.151.181.4 66.151.181.10 70.42.51.30 # Seznam # UA "SeznamBot/1.0 (+http://fulltext.seznam.cz/)" # 212.80.76.87 # Shopwiki # UA "ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot)" 38.98.120.85 8.11.2.19 8.11.2.95 # Singing Fish # UA "asterias/2.0" # UA "Mozilla/4.0 (compatible; MSIE 6.0 compatible; Asterias Crawler v4; +http://www.singingfish.com/help/spider.html; webmaster@singingfish.com); SpiderThread Revision: 1.9" 63.251.10.139 63.251.169.236 64.12.186.194 64.12.186.197 64.12.186.198 64.12.186.199 64.12.186.201 64.12.186.203 64.12.186.206 64.12.186.207 64.12.186.208 64.12.186.209 206.253.222.233 # Sitesearch.ca # UA "Reaper [2.03.10-031204] (http://www.sitesearch.ca/reaper/)" # UA "Helix/1.2 (+http://www.sitesearch.ca/helix/)" 65.110.1.7 65.110.21.171 149.99.7.152 # SitiDi.net # UA "SitiDi.net/SitiDiBot/1.0 (+Have Good Day)" 67.15.42.16 # SmartDesk 128.121.225.20 # Smart Spider 207.16.241 # SmealSearch # UA "SMEALSearch-Bot" 146.186.148.76 # Snap.com # UA "semanticdiscovery/0.2(http://www.semanticdiscovery.com/sd/robot.html)" # UA "semanticdiscovery/0.4(http://www.semanticdiscovery.com/sd/robot.html" # UA "snap.com beta crawler v0" # UA "Snapbot/1.0" # 38.98.19.100 # 38.98.19.101 # 38.98.19.102 # 38.98.19.103 # 38.98.19.104 # 38.98.19.105 # 38.98.19.106 # 38.98.19.107 # 38.98.19.108 # 38.98.19.109 # 38.98.19.110 # 38.98.19.111 # 38.98.19.112 # 38.98.19.113 # 38.98.19.114 # 38.98.19.115 # 38.98.19.116 # 38.98.19.117 # 38.98.19.118 # 38.98.19.121 # 38.98.19.122 # 38.98.19.123 # 38.98.19.124 # 38.98.19.125 # 38.98.19.126 # 38.98.19.66 # 38.98.19.67 # 38.98.19.68 # 38.98.19.69 # 38.98.19.70 # 38.98.19.71 # 38.98.19.75 # 38.98.19.76 # 38.98.19.77 # 38.98.19.79 # 38.98.19.81 # 38.98.19.83 # 38.98.19.85 # 38.98.19.88 # 38.98.19.89 # 38.98.19.90 # 38.98.19.91 # 38.98.19.92 # 38.98.19.93 # 38.98.19.94 # 38.98.19.96 # 38.98.19.97 # 38.98.19.98 # 38.98.19.99 # 64.211.62.5 # 66.234.139.194 # 66.234.139.197 # 66.234.139.198 # 66.234.139.199 # 66.234.139.201 # 66.234.139.202 # 66.234.139.203 # 66.234.139.210 # 66.234.139.211 # 66.234.139.217 # 66.234.139.220 # Sofcom 203.89.255.8 # Sohu # UA "sohu-search" # UA "sohu agent" # UA "sogou spider" 61.135.130.77 61.135.130.78 61.135.130.79 61.135.130.80 61.135.130.100 61.135.131.118 61.135.131.125 61.135.131.128 61.135.131.163 61.135.131.166 61.135.131.171 61.135.131.173 61.135.131.174 61.135.131.207 61.135.131.209 61.135.131.214 61.135.131.230 61.135.131.233 61.135.131.235 61.135.131.237 61.135.131.238 220.181.19.103 220.181.19.164 220.181.19.171 220.181.19.177 220.181.19.65 220.181.19.87 220.181.26.102 220.181.26.106 220.181.26.111 220.181.26.113 220.181.26.121 220.181.26.73 220.181.26.74 # Sonera 194.197.68.46 # speedfind 194.97.8.162 194.97.8.163 192.109.251.26 194.221.132.56 194.221.132.139 # Splat! Search # UA "libwww-perl/5.45" 128.211.213.117 207.44.130.81 207.44.142.84 209.150.128.145 216.71.84.57 216.71.84.212 216.71.187.134 # Sproose # UA "sproose/0.1-alpha (sproose crawler; http://www.sproose.com/bot.html; crawler@sproose.com)" # UA "sproose/0.1 (sproose bot; http://www.sproose.com/bot.html; crawler@sproose.com)" 38.100.225.3 38.100.225.4 38.100.225.5 38.100.225.6 38.100.225.7 38.100.225.8 38.100.225.9 38.100.225.10 38.100.225.11 38.100.225.12 38.100.225.13 38.100.225.14 38.100.225.15 38.100.225.16 38.100.225.17 38.100.225.18 38.100.225.19 38.100.225.20 38.100.225.21 38.100.225.22 38.100.225.23 38.100.225.24 38.100.225.25 38.100.225.26 # Spry Wizard Robot 198.147.135.13 165.121.1.77 165.121.2.77 198.185.1.224 # Spider 194.231.30.15 # SpiderBot 209.203.234.4 207.77.90.17 # SpiderMonkey.ca # UA "Spider_Monkey/7.06 (SpiderMonkey.ca info at http://SpiderMonkey.ca /sm.shtml)" 66.163.18.197 # SquigglebotBot # UA "SquigglebotBot/1.0 http://squigglebot.com" 66.177.183.22 # Suchmaschine.de 195.20.227.67 # SuperSnooper 69.141.14.141 162.33.251.50 207.8.212.162 207.8.212.163 # surfnomore 66.28.248.146 # Sygol # UA "http://www.sygol.com" 81.208.26.55 # Szukacz.pl # UA "Szukacz/1.5 (robot; www.szukacz.pl/jakdzialarobot.html; info@szukacz.pl)" 62.181.185.37 62.181.185.44 193.218.115.6 193.218.115.7 193.218.115.8 193.218.115.81 193.218.115.254 194.181.35.5 194.181.35.6 213.134.142.22 213.134.142.50 # Szukanko.com # UA "NMG Spider/0.3 (szukanko.com)" 217.160.254.242 # Tecnoseek.it 212.97.42.229 # Terrawiz # UA "TerrawizBot/1.0 (+http://www.terrawiz.com/bot.html)" 209.128.80.131 209.128.80.133 209.128.80.136 24.6.176.192 # Thunderstone # UA "T-H-U-N-D-E-R-S-T-O-N-E" # UA "Mozilla/4.0 (compatible; T-H-U-N-D-E-R-S-T-O-N-E)" 63.251.4.43 198.49.220.81 206.183.1.74 207.218.150.79 208.51.0.20 208.51.0.74 208.51.0.79 # Tiscalinet.it 195.130.233.22 195.130.233.30 195.130.233.60 # TitIn 161.53.120.3 # T-Online 212.185.44.13 # Touristlinks # UA "Mozilla/4.0 (GeorgeTheTouristBot;http://www.touristlinks.net)" 81.152.64.189 # Track 194.151.1.60 # Tygo # UA "TygoBot" # UA "TygoProwler (http://www.tygo.com/)" 146.82.72.23 146.82.72.24 # UCSD Crawl 132.239.50.245 # uk-searcher.co.uk # UA "uk-Searcher(HTTP://WWW.UK-SEARCHER.CO.UK)" 81.27.96.248 81.27.99.141 # UKSearchPages # UA "uksearchpages.co.uk" 217.151.96.52 # UKWizz.com # UA "Mackster( http://www.ukwizz.com )" # UA "UKWizz/Nutch-0.8.1 (UKWizz Nutch crawler; http://www.ukwizz.com/)" 195.137.7.76 213.177.232.41 # Updated.com # UA "search.updated.com/0.06 (search.updated.com; http://www.updated.com/docucmentation/bot.html; crawler@updated.com)" # UA "updated/0.1beta (updated.com; http://www.updated.com; crawler@updated.com)" # UA "updated/0.1-alpha (updated crawler; http://www.updated.com; crawler@updated.com)" 38.119.96.100 38.119.96.103 38.119.96.107 38.119.96.110 38.119.96.114 38.119.96.115 38.119.96.116 38.119.96.117 38.119.96.118 38.119.96.119 38.119.96.120 38.119.96.121 # Valkyrie 209.67.119.9 # Verno 133.9.215.72 133.9.215.87 # VerticalMatch.com # UA "VMBot/0.7.2 (VMBot; http://www.VerticalMatch.com/; vmbot@tradedot.com)" 202.83.221.219 210.17.245.180 210.17.245.191 # Vindex 193.172.236.108 193.172.237.17 193.172.236.114 193.172.237.16 193.172.236.8 # vision-search 128.2.206.215 # voila.fr # voila.com # UA "Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot BETA 1.2 (http://www.voila.com/)" 193.252.118.101 193.252.118.102 193.252.118.188 193.252.118.190 193.252.121.229 193.252.148.208 193.252.148.209 193.252.148.51 193.252.149.20 195.101.94.80 195.101.94.81 195.101.94.15 195.101.94.101 195.101.94.166 195.101.94.208 195.101.94.209 209.185.188.207 216.35.76.11 81.52.143.15 81.52.143.16 # W3matter.com 64.95.2.212 # W8 Search 216.104.145.2 216.104.145.160 # Walhello # UA "appie 1.1 (www.walhello.com)" 212.127.141.18 212.58.162.42 212.58.162.78 212.58.169.133 212.58.169.181 213.73.161.41 213.73.177.37 213.73.197.30 213.73.210.224 213.73.211.74 213.73.211.172 213.10.10.117 213.10.10.118 80.60.35.143 81.205.39.64 82.217.42.23 84.104.216.167 84.104.217.36 84.104.217.38 84.104.39.226 # WebBandit Web Spider 199.182.120.206 # Web Core 193.136.20.2 193.136.20.250 # WebCrawler 198.3.99.101 # Webcrawl.net # UA "webcrawl.net" 206.191.49.69 # Web.de 194.45.170.120 # WebFind 65.105.223.11 # WebGather # UA "WebGather 3.0" 61.139.65.222 144.214.122.55 # WebIndexer # UA "WebIndexer/1-dev (Web Indexer; mailto://webindexerv1@yahoo.com; webindexerv1@yahoo.com)" 66.180.173.42 # WebRover # UA "webrover/0.1 (+http://www.webrover.de)" 62.75.193.84 # Websearch.com.au # UA "WebSearch.COM.AU/3.0.1 (The Australian Search Engine; http://WebSearch.COM.AU; Search@WebSearch.COM.AU)" 203.51.46.83 203.51.44.181 # WebsearchUS # UA "MSIE 6.0" 217.73.164.106 # Webseek.de 62.96.181.197 # Webtop 212.111.41.2 212.111.41.33 212.111.41.34 212.111.41.35 212.111.41.36 212.111.41.52 212.111.41.53 212.111.41.151 212.111.41.154 212.111.41.153 212.111.41.152 212.135.14.4 # WebWobot # UA "ScollSpider/2.0 (+http://www.webwobot.com/ScollSpider.php)" 82.43.129.240 # Web Wombat 202.139.99.130 202.139.99.131 202.139.99.131 210.8.18.66 203.9.252.2 # WhatchaSeek # UA "WhatchaBot10 (http://www.whatchaseek.com)" 67.67.130.238 # whatUseek 209.69.255.132 209.69.255.131 209.69.255.160 # WhizBang 63.173.190.2 63.173.190.16 63.173.190.30 63.173.190.152 63.225.238.7 63.225.238.11 216.250.143.106 216.250.143.102 208.1.109.130 # WholeWeb.net 63.140.184.187 192.41.47.46 209.19.244.162 63.140.184.168 63.140.184.171 63.140.184.172 # WhoWhere Robot 205.230.7.23 # Wild Ferret Web Hopper 207.178.193.51 # wise-guys.nl # UA "Mozilla/5.0 (compatible; Vagabondo/2.1; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)" # UA "Bilbo" # UA "Mozilla/4.0 (compatible; Vagabondo/2.3; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)" 194.109.125.201 212.19.205.147 # Witch witch.de 212.172.247.162 # wolong.de 209.78.25.59 # Worldlight # UA "WorldLight" 62.13.25.237 # WORLD SEARCH CENTER 207.153.39.132 207.153.23.8 # World Wide Weasel # UA "WWWeasel Robot v1.00 (http://wwweasel.de)" 80.237.184.66 80.237.202.146 # The World Wide Web Worm 128.138.236.20 # Wotbox # UA "Wotbox/alpha0.5.1 (bot@wotbox.com; http://www.wotbox.com) Java/1.4.1_02" 81.5.184.25 # wwwster # UA "wwwster/1.4 (Beta, mailto:gue@cis.uni-muenchen.de)" 129.187.148.160 129.187.148.161 129.187.148.162 129.187.148.163 129.187.148.164 129.187.148.165 # Xirq.com # UA "xirq/0.1-dev (xirq; http://www.xirq.com; xirq@xirq.com)" 70.86.206.170 # XTRA 203.96.111.201 # Yadows # UA "YadowsCrawler (http://www.yadows.com)" 213.186.46.88 # Yandex # UA "Yandex/1.01.001 (compatible; Win16; H)" # UA "Yandex/1.01.001 (compatible; Win16; I)" # UA "Yandex/1.01.001 (compatible; Win16; P)" # UA "Yandex/1.03.000 (compatible; Win16; M)" # UA "Yandex/1.03.003 (compatible; Win16; D)" # UA "Yandex/1.03.003 (compatible; Win16; D)" # UA "Yandex/2.01.000 (compatible; Win16; Dyatel; C)" # UA "Yandex/2.01.000 (compatible; Win16; Dyatel; D)" # UA "Yandex/2.01.000 (compatible; Win16; Dyatel; N)" # UA "Yandex/2.01.000 (compatible; Win16; Dyatel; Z)" 213.180.193 213.180.194 213.180.206 213.180.207 213.180.210 213.180.216 213.180.217 # Yeehaa 208.197.37.29 # Yodao.com # UA "YodaoBot/1.0 (http://www.yodao.com/help/webmaster/spider/; )" 60.191.80.26 60.191.80.27 60.191.80.28 60.191.80.31 60.191.80.34 60.191.80.35 60.191.80.37 60.191.80.38 60.191.80.39 60.191.80.40 60.191.80.43 60.191.80.44 # Yuntis # UA "Gulper Web Bot 0.2.4 (www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot)" 130.245.134.62 130.245.134.63 130.245.134.64 # Zao - Kototoi Project # UA "Zao/0.1 (http://www.kototoi.org/zao/)" # UA "Zao/0.2 (http://www.kototoi.org/zao/)" # UA "Zao-Crawler" 133.11.36.24 133.11.36.25 133.11.36.26 133.11.36.28 133.11.36.32 133.11.36.34 133.11.36.36 133.11.36.37 133.11.36.38 133.11.36.39 133.11.36.41 133.11.36.42 133.11.36.43 133.11.36.44 133.11.36.45 133.11.36.46 133.11.36.47 133.11.36.48 133.11.36.50 133.11.36.51 133.11.36.52 133.11.36.54 133.11.36.55 # Zibb.com # UA "zibber-v0.1(www.zibb.com/crawler/)" 208.68.136.5 70.42.51.11 # Zippp # UA "ZIPPPCVS/0.24 (ZipppBot/.24; http://www.zippp.net; webmaster@zippp.net)" # UA "ZipppBot/0.11 (ZipppBot; http://www.zippp.net; webmaster@zippp.net)" 65.197.137.32 65.197.137.34 65.197.137.35 65.197.137.37 65.219.130.20 65.219.130.21 65.219.130.240 65.219.130.241 69.20.190.201 # ZoomInfo # UA "NextGenSearchBot 1 (for information visit http://about.zoominfo.com/PublicSite/NextGenSearchBot.asp)" # 67.108.223.130 # zSearch.net 216.55.128.47 # Alexa # UA "ia_archiver" # Has many IPs crawl8-public.alexa.com 209.247.40.99 # Almaden # UA "http://www.almaden.ibm.com/cs/crawler" # IBM research project wfp2.almaden.ibm.com 198.4.83.49 # Cyveillance # UA "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" # A snoop bot to check for copyright/trademark violations 63.148.99.224 - 63.148.99.255 # Curl # UA "curl/7.9.1 (win32) libcurl 7.9.1" # it's a php tool used to grab web pages, etc. # comes from a multitude of IPs # DTSearch # UA "dtSearchSpider" # retail search software # probably coming from many IPs 64.222.18.44 # Eidetica.com # UA "Mozilla/4.7 (compatible; http://eidetica.com/spider)" # Search engine hosting/spidering service idle.eidetica.com 62.58.2.5 # Fantoma # UA "Mozilla/4.0 (fantomBrowser)" # UA "Mozilla/4.0 (stealthBrowser)" # UA "Mozilla/4.0 (cloakBrowser)" # UA "Mozilla/4.0 (fantomCrew Browser)" # UA "multiBlocker browser - IP blocker for Spam, Fraud + Snoop Protection" # UA "multiBlocker browser" # does spidering for promotional purposes and cataloging purposes # also spoofs referring URL in header with link to their website # Girafa.com # UA "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; http://www.girafa.com)" # a client program similar to Alexa 64.210.196.195 64.210.196.198 # Googlebot Fake # UA "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" # This is somebody in China with a bot faking Googlebot's user agent # I've heard reports it's a hacker 211.154.211.209 # Googlebot Fake # UA "Mozilla/4.0 (compatible; MSIE 6.0; Googlebot/2.1 (+http://www.googlebot.com/bot.html); .NET CLR 1.0.3705)" # IP resolves to someplace in New Hampshire 66.94.35.20 # Grub.org # UA "-" # open source spider brain.grub.org 208.128.7.215 # IncyWincy # UA "IncyWincy data gatherer(webmaster@loopimprovements.com,http://www.loopimprovements.com/robot.html)" # appears to be a search engine spider for sale to anybody dsl081-243-066.sfo1.dsl.speakeasy.net 64.81.243.66 64.133.109.250 # Indy Library # UA "Mozilla/3.0 (compatible; Indy Library)" # An email harvester # Probably running from many independent locations/IPs 211.101.236.91 211.101.236.162 212.1.26.100 # Inria.fr # UA "Sqworm/2.9.72-BETA (beta_release; 20010821-737; i686-pc-linux-gnu)" # UA "larbin" (or some variations on larbin) # a French research institute # larbin appears to be a distributed spider 213.97.108.143 63.212.171.171 # Intelliseek # UA "Mozilla/4.7 (compatible; Intelliseek; http://www.intelliseek.com)" # A data mining company 64.158.138.48 # Internetseer.com # UA "sitecheck.internetseer.com" # an uptime verifier 198.139.155.7 198.139.155.32 # Larbin # UA "larbin" # a crawler available under GPL # related to Xyleme # running in a number of independent locations # Lexis-Nexis.com # UA "LNSpiderguy" # snoop bot 198.185.18.207 # LinkWalker # UA "LinkWalker" # looks like a link verifier 209.167.50.22 209.167.50.25 # Lite Bot # UA "Lite Bot 0916b" # Probably a distributed bot used for a variety # of purposes. Does not obey robots.txt 24.126.133.124 # MarkWatch.com # UA "MarkWatch/1.0" # trademark violation investigation bot 204.62.226.36 206.190.171.174 206.190.171.175 # Metacarta.com # UA "flunky" # This bot belongs to an information gathering company # Also known as Bigfoot 66.28.20.194 66.28.44.122 66.28.44.123 66.28.44.125 66.28.68.234 66.28.68.235 66.28.68.236 66.28.68.237 # Microsoft URL control # UA "Microsoft URL Control - 5.01.4319" # VisualBasic tool for grabbing web pages. Probably in use # from multiple independent IPs 195.166.231.3 # NameProtect # UA "NAMEPROTECT" # UA "Mozilla/4.7" # UA "RPT-HTTPClient/0.3-3" # UA "NPBot-1/2.0 (http://www.nameprotect.com/botinfo.html)" # snoop bot to check for trademark violations crawler918.com 12.40.85 12.148.209.196 # NetMechanic # UA "NetMechanic V2.0" 147.208.15.13 216.182.214.7 # NetSpective # UA "WebFilter Robot 1.0" # A web filterer # PicaLoader # UA "PicaLoader 1.0" # A site/picture downloader # Robozilla # UA "Robozilla/1.0" # Link checker for ODP... http://www.dmoz.org 207.200.81.145 # RPT-HTTPClient # UA "RPT-HTTPClient/0.3-3" # A Java library used for spidering # ScoutAbout # UA "ScoutAbout" # is not a SE spider -- it's for sale to anybody zeus.nj.nec.com 138.15.164.9 # SlySearch # UA "TurnitinBot/1.4 http://www.turnitin.com/robot/crawlerinfo.html" # UA "SlySearch" # a bot used to verify plagiarism, etc. # Spidersoft # UA "WebZIP/4.0 (http://www.spidersoft.com)" # web page downloader probably from many independent IPs 212.253.129.11 # Teleport Pro # UA "Teleport Pro/1.28" # a personal bot for windows # operating from many IPs... not worth listing any # Teradex Mapper # UA "(Teradex Mapper; mapper@teradex.com; http://www.teradex.com)" # a bot that belongs to the human reviewed Teradex directory 64.69.79.210 # Tivra # UA "tivraSpider/1.0 (crawler@tivra.com)" # looks like an IBM or AT&T research spider panchma.tivra.com 207.140.168.143 207.140.168.146 # Tracerlock # http://www.tracerlock.com # news monitoring service # UA "libwww-perl/5.47" tracerlock.com 209.61.182.37 # UbiCrawler # UA "UbiCrawler/v0.3beta (http://ubi.imc.pi.cnr.it/projects/ubicrawler/)" # an experimental bot 146.48.78.32 146.48.78.38 # Webclipping.com # UA "Webclipping.com" # a data mining bot 209.73.228.163 209.73.228.167 # Webrank # UA "webrank" # looks like a ranking checker xs4.kso.co.uk 207.235.6.157 # Websquash.com # UA: "websquash.com ( Add Url Robot )" # A fishy engine that belongs to sitereleases.com, a SEO company 66.221.171.1 # websmostlinked.com # UA "lwp-trivial/1.34" # UA "AESOP_com_SpiderMan" # could be a legitimate spider but more likely a hazard for cloakers 64.14.241.54 # Whizbang # UA "ozilla/4.7 (compatible; Whizbang)" # WhizBang is a company that sells a spider to build databases pixnat06.whizbang.com 63.173.190.16 pixnat09.whizbang.com 63.173.190.19 # X-Crawler # UA "TECOMAC-Crawler/0.3" # UA "X-Crawler" # a distributed spider sold by Arexera # Xyleme.com # UA "cosmos/0.8)_(robot@xyleme.com)" # related to Inria.fr # related to Larbin 212.73.246.73 212.73.246.71 # Yahoo.com URL verifiers # UA "Mozilla/4.05" morgue1.corp.yahoo.com 216.145.54.35 hanta.yahoo.com 216.145.50.40 # Zealbot # UA "Mozilla/4.0(compatible; Zealbot 1.0)" # LookSmarts link checker 64.241.242.11 64.241.243.32 64.241.243.65 64.241.243.66 # Zeus Webster # UA "Zeus 60359 Webster Pro V2.9 Win32" # directory builder available retail # # Please, we do NOT allow nonauthorized robots any longer. # User-agent: asterias Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: Black Hole Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BotALot Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: CheeseBot Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: DittoSpyder Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: EroCrawler Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Foobot Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: hloader Disallow: / User-agent: httplib Disallow: / User-agent: humanlinks Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: JennyBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: LexiBot Disallow: / User-agent: libWeb/clsHTTP Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: Mata Hari Disallow: / User-agent: Microsoft URL Control - 5.01.4511 Disallow: / User-agent: Microsoft URL Control - 6.00.8169 Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Mister PiX Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: mozilla/4 Disallow: / User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 95) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 98) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows XP) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows ME) Disallow: / User-agent: mozilla/5 Disallow: / User-agent: NetAnts Disallow: / User-agent: NICErsPRO Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RMA Disallow: / User-agent: SiteSnagger Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: suzuran Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: Telesoft Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: URLy Warning Disallow: / User-agent: VCI Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: Web Image Collector Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: WebCopier Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: WebSauger Disallow: / User-agent: Website Quester Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebZip Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: * Disallow: /ct/dat/