理解http request headers中Referer||User-Agent||Cookie...的意义

本文介绍了HTTP请求头中的关键字段,如Referer、User-Agent和Cookie的作用及意义,并详细解释了它们如何帮助网站跟踪用户行为和维护状态信息。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

  • 缘起

    Python写爬虫,对于AJAX内容,需要在request.get(url, headers)中传入headers,需要了解headers中各项的意义。

  • Request header 请求头

    HTTP header fields are components of the header section of request and response messages in the Hypertext Transfer Protocol (HTTP). They define the operating parameters of an HTTP transaction.

    Standard request fields and Standard response fields

    Referer : This is the address of the previous web page from which a link to the currently requested page was followed.

    User-Agent : The user agent string of the user agent. It is the fields which User agent identification is transmitted. When a software agent operates in a network protocol, it often identifies itself, its application type, operating system, software vendor, or software revision, by submitting a characteristic identification string to its operating peer.

    Cookie : An HTTP cookie previously sent by the server with Set-Cookies.

  • HTTP cookie

    An HTTP cookie(also called web cookie, Internet cookie, brower cookie, or simply cookie) is a small piece of data stored on the user’s computer by the web browser while browsing a website.

    Cookies were designed to be a reliable mechanism for websites to remember stateful information (such as items added in the shopping cart in an online store) or to record the user’s browsing activity(including clicking particular buttons, logging in, or recording which pages were visited in the past). They can also be used to remember pieces of information that user previously entered into form fields, such as names, addresses, passwords, and payment card numbers.

日志:[TASK] ====== CONTINUOUS DOWNLOAD STARTED ====== [INFO] Devices: 8 | Save path: F:/GKD/ [PROCESS] Device: GK2025040001 | Date: 2025-07-01 [DEBUG] Time Parameters: Original Local: 2025-07-01T00:00 - 2025-07-01T23:59:59 Converted Server: 2025-07-01 00:00:00 - 2025-07-01 23:59:59 Raw Time: 2025-07-01 00:00:00 & 2025-07-01 23:59:59 [DEBUG] Final request URL: http://nyzbwlw.com/situation/http/device/bug/getBugData?deviceCode=GK2025040001&startTime=2025-07-01%2000:00:00&endTime=2025-07-01%2023:59:59&size=1000&t=1754026969103 2025-08-01 13:42:49 [Thread-1] DEBUG o.s.web.client.RestTemplate - HTTP GET http://nyzbwlw.com/situation/http/device/bug/getBugData?deviceCode=GK2025040001&startTime=2025-07-01%252000:00:00&endTime=2025-07-01%252023:59:59&size=1000&t=1754026969103 2025-08-01 13:42:49 [Thread-1] DEBUG o.s.web.client.RestTemplate - Accept=[text/plain, application/json, application/*+json, */*] 2025-08-01 13:42:49 [Thread-1] DEBUG o.a.h.c.protocol.RequestAddCookies - CookieSpec selected: default 2025-08-01 13:42:49 [Thread-1] DEBUG o.a.h.c.protocol.RequestAuthCache - Auth cache not set in the context 2025-08-01 13:42:49 [Thread-1] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection request: [route: {}->http://nyzbwlw.com:80][total available: 0; route allocated: 0 of 2; total allocated: 0 of 20] 2025-08-01 13:42:49 [Thread-1] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection leased: [id: 0][route: {}->http://nyzbwlw.com:80][total available: 0; route allocated: 1 of 2; total allocated: 1 of 20] 2025-08-01 13:42:49 [Thread-1] DEBUG o.a.h.impl.execchain.MainClientExec - Opening connection {}->http://nyzbwlw.com:80 2025-08-01 13:42:49 [Thread-1] DEBUG o.a.h.i.c.DefaultHttpClientConnectionOperator - Connecting to nyzbwlw.com/47.92.156.54:80 2025-08-01 13:42:49 [Thread-1] DEBUG o.a.h.i.c.DefaultHttpClientConnectionOperator - Connection established 192.168.2.4:62064<->47.92.156.54:80 2025-08-01 13:42:49 [Thread-1] DEBUG o.a.h.i.c.DefaultManagedHttpClientConnection - http-outgoing-0: set socket timeout to 15000 2025-08-01 13:42:49 [Thread-1] DEBUG o.a.h.impl.execchain.MainClientExec - Executing request GET /situation/http/device/bug/getBugData?deviceCode=GK2025040001&startTime=2025-07-01%252000:00:00&endTime=2025-07-01%252023:59:59&size=1000&t=1754026969103 HTTP/1.1 2025-08-01 13:42:49 [Thread-1] DEBUG o.a.h.impl.execchain.MainClientExec - Target auth state: UNCHALLENGED 2025-08-01 13:42:49 [Thread-1] DEBUG o.a.h.impl.execchain.MainClientExec - Proxy auth state: UNCHALLENGED 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> GET /situation/http/device/bug/getBugData?deviceCode=GK2025040001&startTime=2025-07-01%252000:00:00&endTime=2025-07-01%252023:59:59&size=1000&t=1754026969103 HTTP/1.1 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Accept: application/json, text/plain, */* 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36 Edg/125.0.0.0 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Accept-Language: zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Connection: keep-alive 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Referer: http://nyzbwlw.com/situation/ 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Origin: http://nyzbwlw.com 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Sec-Fetch-Dest: empty 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Sec-Fetch-Mode: cors 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Sec-Fetch-Site: same-origin 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Host: nyzbwlw.com 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Accept-Encoding: gzip,deflate 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "GET /situation/http/device/bug/getBugData?deviceCode=GK2025040001&startTime=2025-07-01%252000:00:00&endTime=2025-07-01%252023:59:59&size=1000&t=1754026969103 HTTP/1.1[\r][\n]" 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Accept: application/json, text/plain, */*[\r][\n]" 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36 Edg/125.0.0.0[\r][\n]" 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Accept-Language: zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6[\r][\n]" 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Connection: keep-alive[\r][\n]" 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Referer: http://nyzbwlw.com/situation/[\r][\n]" 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Origin: http://nyzbwlw.com[\r][\n]" 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Sec-Fetch-Dest: empty[\r][\n]" 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Sec-Fetch-Mode: cors[\r][\n]" 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Sec-Fetch-Site: same-origin[\r][\n]" 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Host: nyzbwlw.com[\r][\n]" 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Accept-Encoding: gzip,deflate[\r][\n]" 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "[\r][\n]" 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "HTTP/1.1 200 [\r][\n]" 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "Server: nginx/1.20.2[\r][\n]" 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "Date: Fri, 01 Aug 2025 05:42:49 GMT[\r][\n]" 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "Content-Type: application/json;charset=UTF-8[\r][\n]" 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "Transfer-Encoding: chunked[\r][\n]" 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "Connection: keep-alive[\r][\n]" 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "Access-Control-Allow-Credentials: true[\r][\n]" 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "Access-Control-Allow-Origin: *[\r][\n]" 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "Vary: Origin[\r][\n]" 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "Access-Control-Expose-Headers: Set-Cookie[\r][\n]" 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "Access-Control-Allow-Methods: POST, GET, PUT, DELETE, OPTIONS[\r][\n]" 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "Set-Cookie: JSESSIONID=852AF1576F877DE9C07BC6B516B6220A; Path=/situation; HttpOnly[\r][\n]" 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "[\r][\n]" 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "41[\r][\n]" 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "{"msg":"[0xe6][0x93][0x8d][0xe4][0xbd][0x9c][0xe6][0x88][0x90][0xe5][0x8a][0x9f][0xef][0xbc][0x81]","code":100,"data":{"count":0,"list":[]}}[\r][\n]" 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "0[\r][\n]" 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "[\r][\n]" 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << HTTP/1.1 200 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << Server: nginx/1.20.2 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << Date: Fri, 01 Aug 2025 05:42:49 GMT 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << Content-Type: application/json;charset=UTF-8 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << Transfer-Encoding: chunked 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << Connection: keep-alive 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << Access-Control-Allow-Credentials: true 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << Access-Control-Allow-Origin: * 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << Vary: Origin 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << Access-Control-Expose-Headers: Set-Cookie 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << Access-Control-Allow-Methods: POST, GET, PUT, DELETE, OPTIONS 2025-08-01 13:42:49 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << Set-Cookie: JSESSIONID=852AF1576F877DE9C07BC6B516B6220A; Path=/situation; HttpOnly 2025-08-01 13:42:49 [Thread-1] DEBUG o.a.h.impl.execchain.MainClientExec - Connection can be kept alive indefinitely 2025-08-01 13:42:49 [Thread-1] DEBUG o.a.h.c.p.ResponseProcessCookies - Cookie accepted [JSESSIONID="852AF1576F877DE9C07BC6B516B6220A", version:0, domain:nyzbwlw.com, path:/situation, expiry:null] 2025-08-01 13:42:49 [Thread-1] DEBUG o.s.web.client.RestTemplate - Response 200 OK 2025-08-01 13:42:49 [Thread-1] DEBUG o.s.web.client.RestTemplate - Reading to [java.lang.String] as "application/json;charset=UTF-8" 2025-08-01 13:42:49 [Thread-1] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection [id: 0][route: {}->http://nyzbwlw.com:80] can be kept alive indefinitely 2025-08-01 13:42:49 [Thread-1] DEBUG o.a.h.i.c.DefaultManagedHttpClientConnection - http-outgoing-0: set socket timeout to 0 2025-08-01 13:42:49 [Thread-1] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection released: [id: 0][route: {}->http://nyzbwlw.com:80][total available: 1; route allocated: 1 of 2; total allocated: 1 of 20] [COOKIE] Saved session cookie: JSESSIONID [DEBUG] Response Status: 200 OK [DEBUG] Raw API response snippet: {"msg":"操作成功!","code":100,"data":{"count":0,"list":[]}} [DEBUG] Parsed API response code: 100 | message: 操作成功! [DEBUG] Found 0 valid image entries [RETRY] Retrying in 2000 ms (attempt 2/3) [DEBUG] Time Parameters: Original Local: 2025-07-01T00:00 - 2025-07-01T23:59:59 Converted Server: 2025-07-01 00:00:00 - 2025-07-01 23:59:59 Raw Time: 2025-07-01 00:00:00 & 2025-07-01 23:59:59 [DEBUG] Final request URL: http://nyzbwlw.com/situation/http/device/bug/getBugData?deviceCode=GK2025040001&startTime=2025-07-01%2000:00:00&endTime=2025-07-01%2023:59:59&size=1000&t=1754026971579 [DEBUG] Sending cookies: JSESSIONID=852AF1576F877DE9C07BC6B516B6220A 2025-08-01 13:42:51 [Thread-1] DEBUG o.s.web.client.RestTemplate - HTTP GET http://nyzbwlw.com/situation/http/device/bug/getBugData?deviceCode=GK2025040001&startTime=2025-07-01%252000:00:00&endTime=2025-07-01%252023:59:59&size=1000&t=1754026971579 2025-08-01 13:42:51 [Thread-1] DEBUG o.s.web.client.RestTemplate - Accept=[text/plain, application/json, application/*+json, */*] 2025-08-01 13:42:51 [Thread-1] DEBUG o.a.h.c.protocol.RequestAddCookies - CookieSpec selected: default 2025-08-01 13:42:51 [Thread-1] DEBUG o.a.h.c.protocol.RequestAddCookies - Cookie [version: 0][name: JSESSIONID][value: 852AF1576F877DE9C07BC6B516B6220A][domain: nyzbwlw.com][path: /situation][expiry: null] match [nyzbwlw.com:80/situation/http/device/bug/getBugData] 2025-08-01 13:42:51 [Thread-1] DEBUG o.a.h.c.protocol.RequestAuthCache - Auth cache not set in the context 2025-08-01 13:42:51 [Thread-1] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection request: [route: {}->http://nyzbwlw.com:80][total available: 1; route allocated: 1 of 2; total allocated: 1 of 20] 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "[read] I/O error: Read timed out" 2025-08-01 13:42:51 [Thread-1] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection leased: [id: 0][route: {}->http://nyzbwlw.com:80][total available: 0; route allocated: 1 of 2; total allocated: 1 of 20] 2025-08-01 13:42:51 [Thread-1] DEBUG o.a.h.i.c.DefaultManagedHttpClientConnection - http-outgoing-0: set socket timeout to 0 2025-08-01 13:42:51 [Thread-1] DEBUG o.a.h.i.c.DefaultManagedHttpClientConnection - http-outgoing-0: set socket timeout to 15000 2025-08-01 13:42:51 [Thread-1] DEBUG o.a.h.impl.execchain.MainClientExec - Executing request GET /situation/http/device/bug/getBugData?deviceCode=GK2025040001&startTime=2025-07-01%252000:00:00&endTime=2025-07-01%252023:59:59&size=1000&t=1754026971579 HTTP/1.1 2025-08-01 13:42:51 [Thread-1] DEBUG o.a.h.impl.execchain.MainClientExec - Target auth state: UNCHALLENGED 2025-08-01 13:42:51 [Thread-1] DEBUG o.a.h.impl.execchain.MainClientExec - Proxy auth state: UNCHALLENGED 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> GET /situation/http/device/bug/getBugData?deviceCode=GK2025040001&startTime=2025-07-01%252000:00:00&endTime=2025-07-01%252023:59:59&size=1000&t=1754026971579 HTTP/1.1 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Accept: application/json, text/plain, */* 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36 Edg/125.0.0.0 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Accept-Language: zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Connection: keep-alive 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Referer: http://nyzbwlw.com/situation/ 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Origin: http://nyzbwlw.com 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Sec-Fetch-Dest: empty 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Sec-Fetch-Mode: cors 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Sec-Fetch-Site: same-origin 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Cookie: JSESSIONID=852AF1576F877DE9C07BC6B516B6220A 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Host: nyzbwlw.com 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Cookie: JSESSIONID=852AF1576F877DE9C07BC6B516B6220A 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Accept-Encoding: gzip,deflate 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "GET /situation/http/device/bug/getBugData?deviceCode=GK2025040001&startTime=2025-07-01%252000:00:00&endTime=2025-07-01%252023:59:59&size=1000&t=1754026971579 HTTP/1.1[\r][\n]" 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Accept: application/json, text/plain, */*[\r][\n]" 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36 Edg/125.0.0.0[\r][\n]" 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Accept-Language: zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6[\r][\n]" 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Connection: keep-alive[\r][\n]" 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Referer: http://nyzbwlw.com/situation/[\r][\n]" 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Origin: http://nyzbwlw.com[\r][\n]" 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Sec-Fetch-Dest: empty[\r][\n]" 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Sec-Fetch-Mode: cors[\r][\n]" 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Sec-Fetch-Site: same-origin[\r][\n]" 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Cookie: JSESSIONID=852AF1576F877DE9C07BC6B516B6220A[\r][\n]" 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Host: nyzbwlw.com[\r][\n]" 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Cookie: JSESSIONID=852AF1576F877DE9C07BC6B516B6220A[\r][\n]" 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Accept-Encoding: gzip,deflate[\r][\n]" 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "[\r][\n]" 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "HTTP/1.1 200 [\r][\n]" 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "Server: nginx/1.20.2[\r][\n]" 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "Date: Fri, 01 Aug 2025 05:42:51 GMT[\r][\n]" 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "Content-Type: application/json;charset=UTF-8[\r][\n]" 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "Transfer-Encoding: chunked[\r][\n]" 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "Connection: keep-alive[\r][\n]" 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "Access-Control-Allow-Credentials: true[\r][\n]" 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "Access-Control-Allow-Origin: *[\r][\n]" 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "Vary: Origin[\r][\n]" 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "Access-Control-Expose-Headers: Set-Cookie[\r][\n]" 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "Access-Control-Allow-Methods: POST, GET, PUT, DELETE, OPTIONS[\r][\n]" 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "[\r][\n]" 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "41[\r][\n]" 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "{"msg":"[0xe6][0x93][0x8d][0xe4][0xbd][0x9c][0xe6][0x88][0x90][0xe5][0x8a][0x9f][0xef][0xbc][0x81]","code":100,"data":{"count":0,"list":[]}}[\r][\n]" 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "0[\r][\n]" 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "[\r][\n]" 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << HTTP/1.1 200 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << Server: nginx/1.20.2 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << Date: Fri, 01 Aug 2025 05:42:51 GMT 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << Content-Type: application/json;charset=UTF-8 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << Transfer-Encoding: chunked 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << Connection: keep-alive 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << Access-Control-Allow-Credentials: true 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << Access-Control-Allow-Origin: * 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << Vary: Origin 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << Access-Control-Expose-Headers: Set-Cookie 2025-08-01 13:42:51 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << Access-Control-Allow-Methods: POST, GET, PUT, DELETE, OPTIONS 2025-08-01 13:42:51 [Thread-1] DEBUG o.a.h.impl.execchain.MainClientExec - Connection can be kept alive indefinitely 2025-08-01 13:42:51 [Thread-1] DEBUG o.s.web.client.RestTemplate - Response 200 OK 2025-08-01 13:42:51 [Thread-1] DEBUG o.s.web.client.RestTemplate - Reading to [java.lang.String] as "application/json;charset=UTF-8" 2025-08-01 13:42:51 [Thread-1] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection [id: 0][route: {}->http://nyzbwlw.com:80] can be kept alive indefinitely 2025-08-01 13:42:51 [Thread-1] DEBUG o.a.h.i.c.DefaultManagedHttpClientConnection - http-outgoing-0: set socket timeout to 0 2025-08-01 13:42:51 [Thread-1] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection released: [id: 0][route: {}->http://nyzbwlw.com:80][total available: 1; route allocated: 1 of 2; total allocated: 1 of 20] [DEBUG] Response Status: 200 OK [DEBUG] Raw API response snippet: {"msg":"操作成功!","code":100,"data":{"count":0,"list":[]}} [DEBUG] Parsed API response code: 100 | message: 操作成功! [DEBUG] Found 0 valid image entries [RETRY] Retrying in 4000 ms (attempt 3/3) [DEBUG] Time Parameters: Original Local: 2025-07-01T00:00 - 2025-07-01T23:59:59 Converted Server: 2025-07-01 00:00:00 - 2025-07-01 23:59:59 Raw Time: 2025-07-01 00:00:00 & 2025-07-01 23:59:59 [DEBUG] Final request URL: http://nyzbwlw.com/situation/http/device/bug/getBugData?deviceCode=GK2025040001&startTime=2025-07-01%2000:00:00&endTime=2025-07-01%2023:59:59&size=1000&t=1754026975716 [DEBUG] Sending cookies: JSESSIONID=852AF1576F877DE9C07BC6B516B6220A 2025-08-01 13:42:55 [Thread-1] DEBUG o.s.web.client.RestTemplate - HTTP GET http://nyzbwlw.com/situation/http/device/bug/getBugData?deviceCode=GK2025040001&startTime=2025-07-01%252000:00:00&endTime=2025-07-01%252023:59:59&size=1000&t=1754026975716 2025-08-01 13:42:55 [Thread-1] DEBUG o.s.web.client.RestTemplate - Accept=[text/plain, application/json, application/*+json, */*] 2025-08-01 13:42:55 [Thread-1] DEBUG o.a.h.c.protocol.RequestAddCookies - CookieSpec selected: default 2025-08-01 13:42:55 [Thread-1] DEBUG o.a.h.c.protocol.RequestAddCookies - Cookie [version: 0][name: JSESSIONID][value: 852AF1576F877DE9C07BC6B516B6220A][domain: nyzbwlw.com][path: /situation][expiry: null] match [nyzbwlw.com:80/situation/http/device/bug/getBugData] 2025-08-01 13:42:55 [Thread-1] DEBUG o.a.h.c.protocol.RequestAuthCache - Auth cache not set in the context 2025-08-01 13:42:55 [Thread-1] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection request: [route: {}->http://nyzbwlw.com:80][total available: 1; route allocated: 1 of 2; total allocated: 1 of 20] 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "[read] I/O error: Read timed out" 2025-08-01 13:42:55 [Thread-1] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection leased: [id: 0][route: {}->http://nyzbwlw.com:80][total available: 0; route allocated: 1 of 2; total allocated: 1 of 20] 2025-08-01 13:42:55 [Thread-1] DEBUG o.a.h.i.c.DefaultManagedHttpClientConnection - http-outgoing-0: set socket timeout to 0 2025-08-01 13:42:55 [Thread-1] DEBUG o.a.h.i.c.DefaultManagedHttpClientConnection - http-outgoing-0: set socket timeout to 15000 2025-08-01 13:42:55 [Thread-1] DEBUG o.a.h.impl.execchain.MainClientExec - Executing request GET /situation/http/device/bug/getBugData?deviceCode=GK2025040001&startTime=2025-07-01%252000:00:00&endTime=2025-07-01%252023:59:59&size=1000&t=1754026975716 HTTP/1.1 2025-08-01 13:42:55 [Thread-1] DEBUG o.a.h.impl.execchain.MainClientExec - Target auth state: UNCHALLENGED 2025-08-01 13:42:55 [Thread-1] DEBUG o.a.h.impl.execchain.MainClientExec - Proxy auth state: UNCHALLENGED 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> GET /situation/http/device/bug/getBugData?deviceCode=GK2025040001&startTime=2025-07-01%252000:00:00&endTime=2025-07-01%252023:59:59&size=1000&t=1754026975716 HTTP/1.1 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Accept: application/json, text/plain, */* 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36 Edg/125.0.0.0 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Accept-Language: zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Connection: keep-alive 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Referer: http://nyzbwlw.com/situation/ 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Origin: http://nyzbwlw.com 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Sec-Fetch-Dest: empty 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Sec-Fetch-Mode: cors 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Sec-Fetch-Site: same-origin 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Cookie: JSESSIONID=852AF1576F877DE9C07BC6B516B6220A 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Host: nyzbwlw.com 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Cookie: JSESSIONID=852AF1576F877DE9C07BC6B516B6220A 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 >> Accept-Encoding: gzip,deflate 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "GET /situation/http/device/bug/getBugData?deviceCode=GK2025040001&startTime=2025-07-01%252000:00:00&endTime=2025-07-01%252023:59:59&size=1000&t=1754026975716 HTTP/1.1[\r][\n]" 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Accept: application/json, text/plain, */*[\r][\n]" 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36 Edg/125.0.0.0[\r][\n]" 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Accept-Language: zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6[\r][\n]" 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Connection: keep-alive[\r][\n]" 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Referer: http://nyzbwlw.com/situation/[\r][\n]" 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Origin: http://nyzbwlw.com[\r][\n]" 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Sec-Fetch-Dest: empty[\r][\n]" 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Sec-Fetch-Mode: cors[\r][\n]" 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Sec-Fetch-Site: same-origin[\r][\n]" 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Cookie: JSESSIONID=852AF1576F877DE9C07BC6B516B6220A[\r][\n]" 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Host: nyzbwlw.com[\r][\n]" 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Cookie: JSESSIONID=852AF1576F877DE9C07BC6B516B6220A[\r][\n]" 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Accept-Encoding: gzip,deflate[\r][\n]" 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 >> "[\r][\n]" 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "HTTP/1.1 200 [\r][\n]" 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "Server: nginx/1.20.2[\r][\n]" 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "Date: Fri, 01 Aug 2025 05:42:55 GMT[\r][\n]" 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "Content-Type: application/json;charset=UTF-8[\r][\n]" 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "Transfer-Encoding: chunked[\r][\n]" 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "Connection: keep-alive[\r][\n]" 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "Access-Control-Allow-Credentials: true[\r][\n]" 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "Access-Control-Allow-Origin: *[\r][\n]" 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "Vary: Origin[\r][\n]" 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "Access-Control-Expose-Headers: Set-Cookie[\r][\n]" 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "Access-Control-Allow-Methods: POST, GET, PUT, DELETE, OPTIONS[\r][\n]" 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "[\r][\n]" 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "41[\r][\n]" 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "{"msg":"[0xe6][0x93][0x8d][0xe4][0xbd][0x9c][0xe6][0x88][0x90][0xe5][0x8a][0x9f][0xef][0xbc][0x81]","code":100,"data":{"count":0,"list":[]}}[\r][\n]" 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "0[\r][\n]" 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.wire - http-outgoing-0 << "[\r][\n]" 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << HTTP/1.1 200 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << Server: nginx/1.20.2 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << Date: Fri, 01 Aug 2025 05:42:55 GMT 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << Content-Type: application/json;charset=UTF-8 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << Transfer-Encoding: chunked 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << Connection: keep-alive 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << Access-Control-Allow-Credentials: true 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << Access-Control-Allow-Origin: * 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << Vary: Origin 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << Access-Control-Expose-Headers: Set-Cookie 2025-08-01 13:42:55 [Thread-1] DEBUG org.apache.http.headers - http-outgoing-0 << Access-Control-Allow-Methods: POST, GET, PUT, DELETE, OPTIONS 2025-08-01 13:42:55 [Thread-1] DEBUG o.a.h.impl.execchain.MainClientExec - Connection can be kept alive indefinitely 2025-08-01 13:42:55 [Thread-1] DEBUG o.s.web.client.RestTemplate - Response 200 OK 2025-08-01 13:42:55 [Thread-1] DEBUG o.s.web.client.RestTemplate - Reading to [java.lang.String] as "application/json;charset=UTF-8" 2025-08-01 13:42:55 [Thread-1] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection [id: 0][route: {}->http://nyzbwlw.com:80] can be kept alive indefinitely 2025-08-01 13:42:55 [Thread-1] DEBUG o.a.h.i.c.DefaultManagedHttpClientConnection - http-outgoing-0: set socket timeout to 0 2025-08-01 13:42:55 [Thread-1] DEBUG o.a.h.i.c.PoolingHttpClientConnectionManager - Connection released: [id: 0][route: {}->http://nyzbwlw.com:80][total available: 1; route allocated: 1 of 2; total allocated: 1 of 20] [DEBUG] Response Status: 200 OK [DEBUG] Raw API response snippet: {"msg":"操作成功!","code":100,"data":{"count":0,"list":[]}} [DEBUG] Parsed API response code: 100 | message: 操作成功! [DEBUG] Found 0 valid image entries [STATS] Device: GK2025040001 | Total: 0 | Success: 0 | Failed: 0 [RESULT] Device: GK2025040001 | Date: 2025-07-01 | Found: 0 | Downloaded: 0 | Failed: 0 代码:package com.example.service; import org.springframework.stereotype.Service; import org.springframework.beans.factory.annotation.Value; import org.springframework.web.client.RestTemplate; import org.springframework.http.ResponseEntity; import org.springframework.http.HttpEntity; import org.springframework.http.HttpHeaders; import org.springframework.http.HttpMethod; import com.fasterxml.jackson.databind.ObjectMapper; import com.fasterxml.jackson.core.type.TypeReference; import java.util.*; import java.time.*; import java.time.format.DateTimeFormatter; import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; import java.io.IOException; import java.util.concurrent.ConcurrentHashMap; import javax.annotation.PostConstruct; import com.example.model.BugApiResponse; import com.example.model.BugResponseData; import com.example.model.BugImageInfo; import java.net.URLEncoder; import java.nio.charset.StandardCharsets; import java.net.InetAddress; import java.util.stream.Collectors; import org.springframework.web.util.UriComponentsBuilder; import java.net.URI; @Service public class BugImageDownloadService { private static final ZoneId SERVER_ZONE = ZoneId.of("Asia/Shanghai"); private static final String API_URL = "http://nyzbwlw.com/situation/http/device/bug/getBugData"; private static final DateTimeFormatter API_DATE_FORMAT = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss"); private static final DateTimeFormatter FILE_DATE_FORMAT = DateTimeFormatter.ofPattern("yyyy-MM-dd"); private static final DateTimeFormatter FILE_TIME_FORMAT = DateTimeFormatter.ofPattern("HH_mm_ss"); private final ConcurrentHashMap<String, LocalDateTime> deviceLastUpdateMap = new ConcurrentHashMap<>(); @Value("${bug.download.deviceCodes:GK2025040001,GK2025040002,GK2025040003,GK2025040004,GK2025040005,GK2025040006,GK2025040007,GK2025040008}") private List<String> deviceCodes; private final Map<String, String> cookies = new ConcurrentHashMap<>(); // 存储每个设备的下一个处理日期 private final ConcurrentHashMap<String, LocalDate> deviceNextDateMap = new ConcurrentHashMap<>(); @Value("${bug.image.save.path:F:/GKD/}") private String savePath; private final RestTemplate restTemplate; private final ObjectMapper objectMapper; public BugImageDownloadService(RestTemplate restTemplate, ObjectMapper objectMapper) { this.restTemplate = restTemplate; this.objectMapper = objectMapper; } @PostConstruct public void init() { // 所有设备从固定起始日期开始 LocalDate startDate = LocalDate.of(2025, 7, 1); for (String deviceCode : deviceCodes) { deviceNextDateMap.put(deviceCode, startDate); } System.out.println("[INIT] Device date tracking initialized. Start date: " + startDate); } /** * 持续下载所有设备的新图片 */ public void downloadNewImagesContinuously() { System.out.println("[TASK] ====== CONTINUOUS DOWNLOAD STARTED ======"); System.out.println("[INFO] Devices: " + deviceCodes.size() + " | Save path: " + savePath); while (true) { try { boolean allDevicesUpToDate = true; // 处理每个设备的历史数据 for (String deviceCode : deviceCodes) { LocalDate currentDate = deviceNextDateMap.get(deviceCode); LocalDate today = LocalDate.now(SERVER_ZONE); if (currentDate.isBefore(today)) { allDevicesUpToDate = false; // 处理该设备当前日期的数据 LocalDateTime startOfDay = currentDate.atStartOfDay(); LocalDateTime endOfDay = currentDate.atTime(23, 59, 59); System.out.printf("[PROCESS] Device: %s | Date: %s%n", deviceCode, currentDate); // 下载该日期的图片 int[] counts = downloadImagesForDevice( deviceCode, startOfDay, endOfDay, false // 标记为历史数据请求 ); // 更新设备的下一个处理日期(明天) deviceNextDateMap.put(deviceCode, currentDate.plusDays(1)); System.out.printf("[RESULT] Device: %s | Date: %s | Found: %d | Downloaded: %d | Failed: %d%n", deviceCode, currentDate, counts[0], counts[1], counts[2]); // 设备间处理间隔 Thread.sleep(2000); } } // 所有设备都处理到最新日期 if (allDevicesUpToDate) { System.out.println("[STATUS] All devices up-to-date. Starting real-time updates..."); // ===== 优化后的实时更新逻辑 ===== for (String deviceCode : deviceCodes) { // 获取上次更新时间(默认当天0点) LocalDateTime lastUpdate = deviceLastUpdateMap.getOrDefault( deviceCode, LocalDate.now(SERVER_ZONE).atStartOfDay() // 使用服务器时区 ); LocalDateTime now = LocalDateTime.now(); System.out.printf("[REALTIME] Device: %s | Time range: %s to %s%n", deviceCode, lastUpdate, now); // 修复点1:添加第四个参数 true 表示实时请求 int[] counts = downloadImagesForDevice( deviceCode, lastUpdate, // 从上次更新时间开始 now, true // 标记为实时请求 ); // 更新最后记录时间(当前轮询结束时间) deviceLastUpdateMap.put(deviceCode, now); System.out.printf("[REALTIME-RESULT] Device: %s | Downloaded: %d | Failed: %d%n", deviceCode, counts[1], counts[2]); // 设备间处理间隔 Thread.sleep(2000); } // ======================== // 等待4h后再次检查 System.out.println("[STATUS] Real-time round completed. Waiting 4 hours..."); Thread.sleep(14_400_000); } } catch (InterruptedException e) { Thread.currentThread().interrupt(); System.err.println("[ERROR] Task interrupted: " + e.getMessage()); break; } catch (Exception e) { System.err.println("[ERROR] Critical error: " + e.getMessage()); try { Thread.sleep(60_000); // 错误后等待1分钟 } catch (InterruptedException ie) { Thread.currentThread().interrupt(); break; } } } } /** * 下载指定设备在指定时间范围内的图片 */ private int[] downloadImagesForDevice(String deviceCode, LocalDateTime startTime, LocalDateTime endTime, boolean isRealtime) { int maxRetries = 3; int retryCount = 0; int waitTime = 2000; // 初始等待时间 2秒 int total = 0; int success = 0; int failed = 0; while (retryCount < maxRetries) { try { // ====== 网络诊断 ====== // ... [网络诊断代码保持不变] ... // ========== 统一时区转换逻辑 ========== ZonedDateTime serverStart = startTime.atZone(ZoneId.systemDefault()) .withZoneSameInstant(SERVER_ZONE); ZonedDateTime serverEnd = endTime.atZone(ZoneId.systemDefault()) .withZoneSameInstant(SERVER_ZONE); // 格式化为API需要的字符串(不进行手动编码) String startStr = serverStart.format(API_DATE_FORMAT); String endStr = serverEnd.format(API_DATE_FORMAT); // 调试日志增强 System.out.println("[DEBUG] Time Parameters:"); System.out.println(" Original Local: " + startTime + " - " + endTime); System.out.println(" Converted Server: " + serverStart.format(API_DATE_FORMAT) + " - " + serverEnd.format(API_DATE_FORMAT)); System.out.println(" Raw Time: " + startStr + " & " + endStr); // 使用Spring的URI构建器确保正确编码 UriComponentsBuilder builder = UriComponentsBuilder.fromHttpUrl(API_URL) .queryParam("deviceCode", deviceCode) .queryParam("startTime", startStr) .queryParam("endTime", endStr) .queryParam("size", 1000) .queryParam("t", System.currentTimeMillis()); URI apiUri = builder.build().encode().toUri(); String apiUrl = apiUri.toString(); System.out.println("[DEBUG] Final request URL: " + apiUrl); // ====== 创建带Cookie的请求头 ====== HttpHeaders headers = new HttpHeaders(); // 添加浏览器级 User-Agent headers.set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36 Edg/125.0.0.0"); // 必须添加的请求头 headers.set("Accept", "application/json, text/plain, */*"); headers.set("Accept-Language", "zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6"); headers.set("Connection", "keep-alive"); headers.set("Referer", "http://nyzbwlw.com/situation/"); // 关键修复:添加 Referer headers.set("Origin", "http://nyzbwlw.com"); headers.set("Sec-Fetch-Dest", "empty"); headers.set("Sec-Fetch-Mode", "cors"); headers.set("Sec-Fetch-Site", "same-origin"); // 添加 Cookie(关键修复:使用持久化 Cookie) if (!cookies.isEmpty()) { String cookieHeader = cookies.entrySet().stream() .map(entry -> entry.getKey() + "=" + entry.getValue()) .collect(Collectors.joining("; ")); headers.add("Cookie", cookieHeader); System.out.println("[DEBUG] Sending cookies: " + cookieHeader); } // 发送请求 ResponseEntity<String> response = restTemplate.exchange( apiUrl, HttpMethod.GET, new HttpEntity<>(headers), String.class); // 保存新Cookie List<String> setCookieHeaders = response.getHeaders().get("Set-Cookie"); if (setCookieHeaders != null) { for (String cookie : setCookieHeaders) { // 提取关键 Cookie(排除过期/路径信息) String[] cookieParts = cookie.split(";")[0].split("="); if (cookieParts.length >= 2) { String cookieName = cookieParts[0].trim(); String cookieValue = cookieParts[1].trim(); cookies.put(cookieName, cookieValue); System.out.println("[COOKIE] Saved session cookie: " + cookieName); } } } // 响应状态日志 System.out.println("[DEBUG] Response Status: " + response.getStatusCodeValue() + " " + response.getStatusCode().getReasonPhrase()); if (response.getStatusCode().is2xxSuccessful() && response.getBody() != null) { String responseBody = response.getBody(); System.out.println("[DEBUG] Raw API response snippet: " + responseBody.substring(0, Math.min(300, responseBody.length())) + (responseBody.length() > 300 ? "..." : "")); // 使用POJO解析响应 BugApiResponse apiResponse = objectMapper.readValue(responseBody, BugApiResponse.class); System.out.printf("[DEBUG] Parsed API response code: %d | message: %s%n", apiResponse.getCode(), apiResponse.getMsg()); if (apiResponse.getCode() == 100) { BugResponseData data = apiResponse.getData(); List<BugImageInfo> bugList = (data != null && data.getList() != null) ? data.getList() : Collections.emptyList(); total = bugList.size(); System.out.printf("[DEBUG] Found %d valid image entries%n", total); // 如果有数据,处理并返回 if (total > 0) { for (BugImageInfo bug : bugList) { String pictureUrl = bug.getPictureUrl(); String collectionTime = bug.getCollectionTime(); if (pictureUrl != null && !pictureUrl.isEmpty() && collectionTime != null && !collectionTime.isEmpty()) { System.out.printf("[PROCESS] Downloading: %s | %s%n", collectionTime, pictureUrl); if (downloadImage(deviceCode, pictureUrl, collectionTime)) { success++; } else { failed++; } } else { System.err.printf("[WARN] Invalid data: URL=%s | Time=%s%n", pictureUrl, collectionTime); failed++; } } return new int[]{total, success, failed}; } } else { System.err.println("[API-ERROR] Device: " + deviceCode + " | Code: " + apiResponse.getCode()); } } } catch (Exception e) { System.err.println("[DOWNLOAD-ERROR] Device: " + deviceCode + " | Error: " + e.getMessage()); e.printStackTrace(); } // 如果没有数据或出错,等待后重试 retryCount++; if (retryCount < maxRetries) { try { System.out.printf("[RETRY] Retrying in %d ms (attempt %d/%d)%n", waitTime, retryCount + 1, maxRetries); Thread.sleep(waitTime); waitTime *= 2; // 指数退避 } catch (InterruptedException ie) { Thread.currentThread().interrupt(); break; } } } System.out.printf("[STATS] Device: %s | Total: %d | Success: %d | Failed: %d%n", deviceCode, total, success, failed); return new int[]{total, success, failed}; } private boolean downloadImage(String deviceCode, String imageUrl, String collectionTime) { try { LocalDateTime dateTime = LocalDateTime.parse(collectionTime, API_DATE_FORMAT); // 创建保存路径 Path saveDir = Paths.get(savePath, deviceCode, dateTime.format(FILE_DATE_FORMAT)); if (!Files.exists(saveDir)) { Files.createDirectories(saveDir); } // 生成文件名 String fileName = dateTime.format(FILE_TIME_FORMAT) + ".jpg"; Path filePath = saveDir.resolve(fileName); // 跳过已存在文件 if (Files.exists(filePath)) { return true; } // 下载图片 byte[] imageBytes = restTemplate.getForObject(imageUrl, byte[].class); if (imageBytes != null && imageBytes.length > 0) { Files.write(filePath, imageBytes); return true; } } catch (Exception e) { System.err.println("[IMAGE-ERROR] URL: " + imageUrl + " | Error: " + e.getMessage()); } return false; } }解决请求http://nyzbwlw.com/situation/http/device/bug/getBugData?deviceCode=GK2025040006&startTime=2025-07-01%252000:00:00&endTime=2025-07-01%252023:59:59&size=1000&t=1754027017557这个api返回count为0,在浏览器中输入这个url返回13条有效数据{"msg":"操作成功!","code":100,"data":{"count":13,"list":[{"eqName":"GK-7","collectionTime":"2025-07-04 05:28:09","monitorId":250704052841640,"exhibitionPath":"","num":"0","pictureUrl":"https://zzoukeqi.oss-cn-beijing.aliyuncs.com/situations/bugImgs/GK2025040007/2025-07-04/250704052841622749.jpg","pictureAnalysis":{}}
最新发布
08-02
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值