From 1d967a4d423c0571a86d319ac16d704bbf29e51a Mon Sep 17 00:00:00 2001 From: Tatsuya Kinoshita Date: Sat, 19 Dec 2020 13:20:49 +0900 Subject: Add examples of siteconf, set user_agent to Googlebot for Twitter --- doc/README.siteconf | 6 ++++++ 1 file changed, 6 insertions(+) (limited to 'doc') diff --git a/doc/README.siteconf b/doc/README.siteconf index 0369926..8514edf 100644 --- a/doc/README.siteconf +++ b/doc/README.siteconf @@ -47,6 +47,12 @@ user_agent "Lynx/2.8.8dev.3 libwww-FM/2.14 SSL-MM/1.4.1" Tell Google we're actually Lynx. (So they send us a text-browser friendly results page.) +url m!^https?://([a-z]+\.)?twitter\.com/! +user_agent "Googlebot/2.1" + +Tell Twitter we're actually Googlebot. (So they send us a page without +rejection of a JavaScript disabled browser.) + ===== Regular expressions notes ===== Following expressions are all equivalent: -- cgit v1.2.3