Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.039830012s
FAIL (2025-07-27 00:28:26 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
d=7BMKAKFYJSGBTDB64PQSNQLQCUDX64OOJJGAF3RBAYAERIZ3VNXA, seg_pos={0 0}, upload=2392]: Uploading piece...
Jul 27 00:27:26.471231100 (2h35m42.252164508s) [noise=true , address=77.90.31.17:4792 , piece_id=I2EFY5REBRIIWNQYGBT5PETJ4PVPUQPQFL27TG23PHOTNI7L4T4Q, seg_pos={0 0}, upload=2392]: Uploading piece...
Jul 27 00:27:26.471356473 (2h35m42.25228986s) [noise=true , address=104.36.82.54:4163 , piece_id=22FA4PAWJB4FQJ43NSJRN3K6UA6PCDK7CKFHPGGH6TDOFAF7YOXQ, seg_pos={0 0}, upload=2392]: Uploading piece...
Jul 27 00:27:26.471772528 (2h35m42.252705915s) [noise=true , address=64.49.13.139:4478 , piece_id=7OUHO2VKK4ZIDDQSBPDEG7KEBQP7GO6J2HNTF7N5Z3O22TTRUR3Q, seg_pos={0 0}, upload=2392]: Uploading piece...
Jul 27 00:27:26.471962598 (2h35m42.252895985s) [noise=true , address=104.36.82.52:4097 , piece_id=43MACCM5RLSY3IXHPFFYNAKJIQY45E6Z24N7TISGAEQSKHZPEVJA, seg_pos={0 0}, upload=2392]: Uploading piece...
Jul 27 00:27:26.472268991 (2h35m42.253202376s) [noise=true , address=79.127.205.245:4248 , piece_id=S25F3LV5UVV6UQRB27OSB76OAFVJJCQDC264D5SKIVFIH6B57KEQ, seg_pos={0 0}, upload=2392]: Uploading piece...
Jul 27 00:27:26.472509494 (2h35m42.253442871s) [noise=true , address=79.127.220.101:4041 , piece_id=MTYQKW3CLES2OMI5WQ3EBYUANNU7T73MPHR57FB7JNH6EBRGRODA, seg_pos={0 0}, upload=2392]: Uploading piece...
Jul 27 00:27:26.473024889 (2h35m42.253958278s) [noise=true , address=79.127.205.246:4278 , piece_id=JVCYSTU6YKUOPIWYBA2NTVNAI4TTT6N6JMLVCVTR6MM22TDZ5NTA, seg_pos={0 0}, upload=2392]: Uploading piece...
Jul 27 00:27:26.473104285 (2h35m42.254037675s) [noise=true , address=79.127.219.38:4006 , piece_id=FMHSWNMCOV5IM2XHDKQGULEJQXBGO6QXAW2GS6L6Y7JJ2TZ653DQ, seg_pos={0 0}, upload=2392]: Uploading piece...
Jul 27 00:27:26.473542915 (2h35m42.254476305s) [noise=true , address=104.36.82.46:4507 , piece_id=SHIBRAZNU65Q5EXKZNJZZ6YT4OJPE2OZOQJH6MUD3EHBJEX4LACA, seg_pos={0 0}, upload=2392]: Uploading piece...
Jul 27 00:27:26.473570465 (2h35m42.254503849s) [noise=true , address=156.146.43.225:4461 , piece_id=B5K343ACIE27RPLTJIVTVE3DWVMRWJEOHWFHBNXKN7A4J3RS2L7Q, seg_pos={0 0}, upload=2392]: Uploading piece...
Jul 27 00:27:26.473959622 (2h35m42.25489301s) [noise=true , address=104.36.82.55:4202 , piece_id=NKVHLWTADQ6ALYP2Z5KBTRGY4YSC272YQXLOUTVWDRUFEYUCSKDA, seg_pos={0 0}, upload=2392]: Uploading piece...
Jul 27 00:27:26.474026753 (2h35m42.254960137s) [noise=true , address=104.36.82.47:4545 , piece_id=5HQAGGCDEHVM52TSNSX3O35ELQYNUO7DKHSJLWITZK4Q3KT7ZB6Q, seg_pos={0 0}, upload=2392]: Uploading piece...
Jul 27 00:27:26.474445495 (2h35m42.255378879s) [noise=true , address=104.36.82.56:4244 , piece_id=U2EIQHP5VR262NR53LOGXVQ7FOAJ5LYWDWFMB5YJRVIIXLXIKNMQ, seg_pos={0 0}, upload=2392]: Uploading piece...
Jul 27 00:27:26.474862688 (2h35m42.25579607s) [noise=true , address=77.90.31.13:4658 , piece_id=SKR62SSGFEGH43T2UECLP2HFAYRBNMNKTFMI53OKXB63JGGHJ74A, seg_pos={0 0}, upload=2392]: Uploading piece...
Jul 27 00:27:26.474873122 (2h35m42.255807289s) [noise=true , address=79.127.205.233:4642 , piece_id=WGKZBMVCFZH5SCCYCVDZYMBGLM2LW33QK45VYDCUXIYWMZUKVVHQ, seg_pos={0 0}, upload=2392]: Uploading piece...
Jul 27 00:27:26.478975067 (2h35m42.259908436s) [noise=true , address=79.127.220.97:4361 , piece_id=TXQYNILTXIKK4NAJ65EQ7JB45SS7PKYP7M5OZ5U2RYPVAWZ4476Q, seg_pos={0 0}, upload=2392]: Uploading piece...
Jul 27 00:27:26.480027718 (2h35m42.260961097s) [noise=true , address=104.36.82.11:4143 , piece_id=VWDZRME2V7OYFIK5EM7FPNHE2RQ4ALAZFEWOOWNTUZ5AGNFAPQMQ, seg_pos={0 0}, upload=2392]: Uploading piece...
Jul 27 00:27:26.520474031 (2h35m42.301407417s) [noise=true , address=77.90.31.13:4658 , piece_id=SKR62SSGFEGH43T2UECLP2HFAYRBNMNKTFMI53OKXB63JGGHJ74A, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: storage node overloaded, request limit: 2000
Jul 27 00:27:26.554999884 (2h35m42.335933262s) [upload=2392]: Next returned nil segment
Jul 27 00:27:26.555095212 (2h35m42.336028598s) [upload=2392]: Waiting for error group managing segments...
Jul 27 00:27:26.559224449 (2h35m42.340157816s) [noise=true , address=79.127.219.38:4006 , piece_id=FMHSWNMCOV5IM2XHDKQGULEJQXBGO6QXAW2GS6L6Y7JJ2TZ653DQ, seg_pos={0 0}, upload=2392]: Done uploading piece. err: <nil>
Jul 27 00:27:26.575349290 (2h35m42.356282673s) [noise=true , address=79.127.220.101:4041 , piece_id=MTYQKW3CLES2OMI5WQ3EBYUANNU7T73MPHR57FB7JNH6EBRGRODA, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:123G35C5bhcQpSSGRtKGo78ZfxNj6bZatY3kCrdEu1kMyjV5Yiw, address:79.127.220.101:4041): protocol: storage node overloaded, request limit: 2000
Jul 27 00:27:26.626795101 (2h35m42.407728471s) [noise=true , address=156.146.43.225:4461 , piece_id=B5K343ACIE27RPLTJIVTVE3DWVMRWJEOHWFHBNXKN7A4J3RS2L7Q, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:1D4k7wnNtivVkHko5CSBuZH6zqMuFyfFfshkZwEPqnmpLads49, address:156.146.43.225:4461): protocol: storage node overloaded, request limit: 2000
Jul 27 00:27:27.001628703 (2h35m42.782562102s) [noise=true , address=79.127.220.97:4361 , piece_id=TXQYNILTXIKK4NAJ65EQ7JB45SS7PKYP7M5OZ5U2RYPVAWZ4476Q, seg_pos={0 0}, upload=2392]: Done uploading piece. err: <nil>
Jul 27 00:27:27.061928944 (2h35m42.842862315s) [noise=true , address=79.127.205.235:4679 , piece_id=7BMKAKFYJSGBTDB64PQSNQLQCUDX64OOJJGAF3RBAYAERIZ3VNXA, seg_pos={0 0}, upload=2392]: Done uploading piece. err: <nil>
Jul 27 00:27:27.096086113 (2h35m42.877019476s) [noise=true , address=79.127.205.233:4642 , piece_id=WGKZBMVCFZH5SCCYCVDZYMBGLM2LW33QK45VYDCUXIYWMZUKVVHQ, seg_pos={0 0}, upload=2392]: Done uploading piece. err: <nil>
Jul 27 00:27:27.703404942 (2h35m43.484338304s) [noise=true , address=79.127.205.245:4248 , piece_id=S25F3LV5UVV6UQRB27OSB76OAFVJJCQDC264D5SKIVFIH6B57KEQ, seg_pos={0 0}, upload=2392]: Done uploading piece. err: <nil>
Jul 27 00:27:29.802564754 (2h35m45.583498125s) [noise=true , address=79.127.205.246:4278 , piece_id=JVCYSTU6YKUOPIWYBA2NTVNAI4TTT6N6JMLVCVTR6MM22TDZ5NTA, seg_pos={0 0}, upload=2392]: Done uploading piece. err: <nil>
Jul 27 00:27:33.581878489 (2h35m49.362811854s) [noise=true , address=64.49.13.139:4478 , piece_id=7OUHO2VKK4ZIDDQSBPDEG7KEBQP7GO6J2HNTF7N5Z3O22TTRUR3Q, seg_pos={0 0}, upload=2392]: Done uploading piece. err: <nil>
Jul 27 00:28:26.387999475 (2h36m42.168932852s) [noise=true , address=104.36.82.56:4244 , piece_id=U2EIQHP5VR262NR53LOGXVQ7FOAJ5LYWDWFMB5YJRVIIXLXIKNMQ, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:12S1wh3PFQLPosjWmPs5iLfm4S5J6abXvoCfkhYz3KdvXCTaPRu, address:104.36.82.56:4244): protocol: write tcp 159.223.143.39:34230->104.36.82.56:4244: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:28:26.388167306 (2h36m42.169100672s) [noise=true , address=77.90.31.17:4792 , piece_id=I2EFY5REBRIIWNQYGBT5PETJ4PVPUQPQFL27TG23PHOTNI7L4T4Q, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:12SPFgYZnbwjEupjaghfs61qwFM678KNPwpDmRNqV1j2opNgZCN, address:77.90.31.17:4792): protocol: context deadline exceeded; context deadline exceeded
Jul 27 00:28:26.388340549 (2h36m42.16927393s) [noise=true , address=104.36.82.47:4545 , piece_id=5HQAGGCDEHVM52TSNSX3O35ELQYNUO7DKHSJLWITZK4Q3KT7ZB6Q, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: context deadline exceeded; context deadline exceeded
Jul 27 00:28:26.388591403 (2h36m42.169524782s) [noise=true , address=104.36.82.54:4163 , piece_id=22FA4PAWJB4FQJ43NSJRN3K6UA6PCDK7CKFHPGGH6TDOFAF7YOXQ, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:104.36.82.54:4163): protocol: rpc: dial tcp 104.36.82.54:4163: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:28:26.388706433 (2h36m42.169639802s) [noise=true , address=104.36.82.52:4097 , piece_id=43MACCM5RLSY3IXHPFFYNAKJIQY45E6Z24N7TISGAEQSKHZPEVJA, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:1sMhTktqf2H4aZSyFV3m9Z6eMVxL2U52bvUHVHUh2gUCVp32AM, address:104.36.82.52:4097): protocol: context deadline exceeded; context deadline exceeded
Jul 27 00:28:26.388810431 (2h36m42.169743798s) [noise=true , address=104.36.82.55:4202 , piece_id=NKVHLWTADQ6ALYP2Z5KBTRGY4YSC272YQXLOUTVWDRUFEYUCSKDA, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:104.36.82.55:4202): protocol: context deadline exceeded; context deadline exceeded
Jul 27 00:28:26.389041442 (2h36m42.169974821s) [noise=true , address=104.36.82.46:4507 , piece_id=SHIBRAZNU65Q5EXKZNJZZ6YT4OJPE2OZOQJH6MUD3EHBJEX4LACA, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R, address:104.36.82.46:4507): protocol: context deadline exceeded; context deadline exceeded
Jul 27 00:28:26.389138404 (2h36m42.17007178s) [noise=true , address=104.36.82.11:4143 , piece_id=VWDZRME2V7OYFIK5EM7FPNHE2RQ4ALAZFEWOOWNTUZ5AGNFAPQMQ, seg_pos={0 0}, upload=2392]: Done uploading piece. err: ecclient: upload failed (node:12ipThd6PwZUpCXy3ML4qqMd9dmRLmxKjVNCNoT5wCg1szdWp8U, address:104.36.82.11:4143): protocol: context deadline exceeded; context deadline exceeded
Jul 27 00:28:26.389177161 (2h36m42.17011055s) [seg_pos={0 0}, upload=2392]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Jul 27 00:28:26.424005881 (2h36m42.204939263s) [upload=2392]: Done uploading segments...
Jul 27 00:28:26.424046045 (2h36m42.204979424s) [upload=2392]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 1m0.039877826s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [.....X.X........X...XX...............................X......] duration:
42.550820116s
FAIL (2025-07-27 00:28:24 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 42.550835699s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [.........X..X......X.XX....X...X.................XX....X....] duration:
2.315862232s
FAIL (2025-07-27 00:28:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.315890827s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [..........X..X......X.XX....X...X.................XX....X...] duration:
29.763447171s
FAIL (2025-07-27 00:28:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 29.763469865s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
40 out of 60 is failed (threshold=3) [XX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX...] duration:
529.958679ms
FAIL (2025-07-27 00:27:50 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
5.226875599 (2h33m1.007808984s) [noise=true , address=104.36.82.4:4801 , piece_id=AARTJBSLFRFJNTY64JVS7C4I5WOW3PMUDP3GSCJ4WDDOMZBHJQUQ, seg_pos={0 0}, upload=2341]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 159.223.143.39:35016->104.36.82.4:4801: use of closed network connection; context deadline exceeded
Jul 27 00:24:45.226924742 (2h33m1.007858123s) [seg_pos={0 0}, upload=2341]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 27 00:24:45.260403710 (2h33m1.041337086s) [upload=2341]: Done uploading segments...
Jul 27 00:24:45.260436570 (2h33m1.041369954s) [upload=2341]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Jul 27 00:25:13.130330845 (2h33m28.911264233s) [upload=2358]: Starting upload
Jul 27 00:25:13.130427709 (2h33m28.911361097s) [upload=2358]: Done starting upload
Jul 27 00:25:13.130504194 (2h33m28.911437572s) [upload=2358]: Uploading segments...
Jul 27 00:25:13.130768741 (2h33m28.911702127s) [upload=2358]: Got next segment. Inline: false
Jul 27 00:25:13.130825048 (2h33m28.911758432s) [upload=2358]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 27 00:25:13.130832706 (2h33m28.911766093s) [upload=2358]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 27 00:25:13.156703675 (2h33m28.937637054s) [seg_pos={0 0}, upload=2358]: Begin upload segment...
Jul 27 00:25:13.156790074 (2h33m28.937723451s) [seg_pos={0 0}, upload=2358]: Done begin upload segment.
Jul 27 00:25:13.156839061 (2h33m28.937772446s) [noise=true , address=45.91.116.8:4048 , piece_id=KBWTGJIF2HUAI44ZNOGFICQMX3LQB3TDNMEIMEUZLKPCC26KBVKQ, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.156878026 (2h33m28.937811413s) [noise=true , address=77.90.31.17:4772 , piece_id=4N2LFDTN7S2YMUYO5L3RPZM7G2WXNM7FZWHEYARKUUEJDPUALQRA, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.157292455 (2h33m28.938225837s) [noise=true , address=77.90.31.10:4518 , piece_id=FDKCHHU7HLMMD72YR6FBQL3SNY3KWC5ROF5C3L4CHYVXMWI52ICQ, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.157494201 (2h33m28.938427581s) [noise=true , address=77.90.31.16:4733 , piece_id=CHIV4NLYUQSUYDARJQBR5O7VNMZMACAPP6CXLSMS6E3GWADVPHMQ, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.157651501 (2h33m28.938584887s) [noise=true , address=45.91.116.10:4081 , piece_id=YJZ54OWAQDBBKU7PCKLO5JMS6P3WIVMJM3G6GW4726MHIIGQPLZQ, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.158080485 (2h33m28.939013869s) [noise=true , address=45.91.116.6:4973 , piece_id=PEBPYGY7NDBWIXKTO6RBZJ2JMXDOSHRN7PZSOCQDJMYFGL4MQPKQ, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.158127600 (2h33m28.939060987s) [noise=true , address=77.90.31.21:4878 , piece_id=XR4HBBDT4QA44YMR5XCE3DBNZXUKMTMN2BLVX3MS6P2SFASEUL3Q, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.158413346 (2h33m28.939346724s) [noise=true , address=77.90.31.15:4699 , piece_id=4AFXJPKKDAXBWYVP2UOUYEMXTKINT6TVE7RU3ASWQBDFUHIZEOCA, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.158633735 (2h33m28.939567118s) [noise=true , address=77.90.31.14:4663 , piece_id=BZDQHDA2WG7I7KZPD3VB4PSGLRYTLKX5JPXJK3TWVMR6U4A7TCPQ, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.158790022 (2h33m28.939723394s) [noise=true , address=77.90.31.18:4807 , piece_id=VOLMX5A4ZPF6WBTMC4GOQFJCUVTB27CH3A5KYRPGUHUWGSRUIHAA, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.159238365 (2h33m28.940171753s) [noise=true , address=45.91.116.13:4153 , piece_id=A4YC7563QYYSR664U2SG5472GSECXWTZTYIBBZHIR22N35AG7U4Q, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.160717151 (2h33m28.941650518s) [noise=true , address=45.91.116.14:4191 , piece_id=HEQI2NLXJ462RJYGOHXPEUFVFXUIMLMWKRI5MKELR7PZHATNKH4A, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.161474674 (2h33m28.942408056s) [noise=true , address=104.36.82.9:4007 , piece_id=YZTTC6XE6LDSWOB5CTTZWRR35UCYOLLBGIXOHYBJVG7G5EZ5EM7A, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.162300982 (2h33m28.943234369s) [noise=true , address=45.91.116.4:4937 , piece_id=MIRJTIN2KP5QR5UPO7QT5DQBMFQY7YO56MVTXDHS54AGLWW5VQHA, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.163746259 (2h33m28.94467965s) [noise=true , address=45.91.116.8:4047 , piece_id=AKYFH3HPWUFIGRCKPSNF5KBMUS57AFU3VSD46KXJJD3KN23PEA7Q, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.164391211 (2h33m28.945324597s) [noise=true , address=104.36.82.4:4802 , piece_id=Y4JKVVI6JBN72UV5FK3QHITG3QN4PXXX34N2NCIKFA2VHW5FX3FA, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.286660997 (2h33m29.067594379s) [upload=2358]: Next returned nil segment
Jul 27 00:25:13.286684184 (2h33m29.067617547s) [upload=2358]: Waiting for error group managing segments...
Jul 27 00:25:13.292602587 (2h33m29.073535969s) [noise=true , address=45.91.116.10:4081 , piece_id=YJZ54OWAQDBBKU7PCKLO5JMS6P3WIVMJM3G6GW4726MHIIGQPLZQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.292748204 (2h33m29.073681574s) [noise=true , address=45.91.116.4:4937 , piece_id=MIRJTIN2KP5QR5UPO7QT5DQBMFQY7YO56MVTXDHS54AGLWW5VQHA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.293326557 (2h33m29.074259939s) [noise=true , address=45.91.116.8:4047 , piece_id=AKYFH3HPWUFIGRCKPSNF5KBMUS57AFU3VSD46KXJJD3KN23PEA7Q, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.293361832 (2h33m29.074295202s) [noise=true , address=77.90.31.21:4878 , piece_id=XR4HBBDT4QA44YMR5XCE3DBNZXUKMTMN2BLVX3MS6P2SFASEUL3Q, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.294304357 (2h33m29.075237733s) [noise=true , address=77.90.31.14:4663 , piece_id=BZDQHDA2WG7I7KZPD3VB4PSGLRYTLKX5JPXJK3TWVMR6U4A7TCPQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.295589750 (2h33m29.076523117s) [noise=true , address=45.91.116.8:4048 , piece_id=KBWTGJIF2HUAI44ZNOGFICQMX3LQB3TDNMEIMEUZLKPCC26KBVKQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.295634225 (2h33m29.076567588s) [noise=true , address=77.90.31.18:4807 , piece_id=VOLMX5A4ZPF6WBTMC4GOQFJCUVTB27CH3A5KYRPGUHUWGSRUIHAA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.296581902 (2h33m29.077515282s) [noise=true , address=45.91.116.13:4153 , piece_id=A4YC7563QYYSR664U2SG5472GSECXWTZTYIBBZHIR22N35AG7U4Q, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.297185145 (2h33m29.078118508s) [noise=true , address=45.91.116.6:4973 , piece_id=PEBPYGY7NDBWIXKTO6RBZJ2JMXDOSHRN7PZSOCQDJMYFGL4MQPKQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.325234980 (2h33m29.106168359s) [noise=true , address=77.90.31.15:4699 , piece_id=4AFXJPKKDAXBWYVP2UOUYEMXTKINT6TVE7RU3ASWQBDFUHIZEOCA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.360841866 (2h33m29.141775245s) [noise=true , address=45.91.116.14:4191 , piece_id=HEQI2NLXJ462RJYGOHXPEUFVFXUIMLMWKRI5MKELR7PZHATNKH4A, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:43.100924665 (2h33m58.88185803s) [noise=true , address=77.90.31.17:4772 , piece_id=4N2LFDTN7S2YMUYO5L3RPZM7G2WXNM7FZWHEYARKUUEJDPUALQRA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 159.223.143.39:56852->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:25:43.101183051 (2h33m58.882116415s) [noise=true , address=104.36.82.4:4802 , piece_id=Y4JKVVI6JBN72UV5FK3QHITG3QN4PXXX34N2NCIKFA2VHW5FX3FA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: context deadline exceeded; context deadline exceeded
Jul 27 00:25:43.101557492 (2h33m58.882490858s) [noise=true , address=77.90.31.10:4518 , piece_id=FDKCHHU7HLMMD72YR6FBQL3SNY3KWC5ROF5C3L4CHYVXMWI52ICQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:25:43.101690637 (2h33m58.882624007s) [noise=true , address=77.90.31.16:4733 , piece_id=CHIV4NLYUQSUYDARJQBR5O7VNMZMACAPP6CXLSMS6E3GWADVPHMQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded
Jul 27 00:25:43.101784731 (2h33m58.882718095s) [noise=true , address=104.36.82.9:4007 , piece_id=YZTTC6XE6LDSWOB5CTTZWRR35UCYOLLBGIXOHYBJVG7G5EZ5EM7A, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 159.223.143.39:54568->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:25:43.101803464 (2h33m58.882736842s) [seg_pos={0 0}, upload=2358]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 27 00:25:43.126349574 (2h33m58.90728296s) [upload=2358]: Done uploading segments...
Jul 27 00:25:43.126382341 (2h33m58.907315726s) [upload=2358]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 530.021974ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
205.808024ms
FAIL (2025-07-27 00:27:43 UTC)
orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K
Duration: 205.833785ms
core
uplink
us1
vivint
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [......X.X........X...XX...............................X.....] duration:
59.192108616s
FAIL (2025-07-27 00:27:42 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 59.192127209s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
39 out of 60 is failed (threshold=3) [XXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XXXX] duration:
30.025111521s
FAIL (2025-07-27 00:27:31 UTC)
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 30.025131047s
core
uplink
us1
vivint
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
41 out of 60 is failed (threshold=3) [XXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX..] duration:
2.56986838s
FAIL (2025-07-27 00:27:30 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
5.226875599 (2h33m1.007808984s) [noise=true , address=104.36.82.4:4801 , piece_id=AARTJBSLFRFJNTY64JVS7C4I5WOW3PMUDP3GSCJ4WDDOMZBHJQUQ, seg_pos={0 0}, upload=2341]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 159.223.143.39:35016->104.36.82.4:4801: use of closed network connection; context deadline exceeded
Jul 27 00:24:45.226924742 (2h33m1.007858123s) [seg_pos={0 0}, upload=2341]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 27 00:24:45.260403710 (2h33m1.041337086s) [upload=2341]: Done uploading segments...
Jul 27 00:24:45.260436570 (2h33m1.041369954s) [upload=2341]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Jul 27 00:25:13.130330845 (2h33m28.911264233s) [upload=2358]: Starting upload
Jul 27 00:25:13.130427709 (2h33m28.911361097s) [upload=2358]: Done starting upload
Jul 27 00:25:13.130504194 (2h33m28.911437572s) [upload=2358]: Uploading segments...
Jul 27 00:25:13.130768741 (2h33m28.911702127s) [upload=2358]: Got next segment. Inline: false
Jul 27 00:25:13.130825048 (2h33m28.911758432s) [upload=2358]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 27 00:25:13.130832706 (2h33m28.911766093s) [upload=2358]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 27 00:25:13.156703675 (2h33m28.937637054s) [seg_pos={0 0}, upload=2358]: Begin upload segment...
Jul 27 00:25:13.156790074 (2h33m28.937723451s) [seg_pos={0 0}, upload=2358]: Done begin upload segment.
Jul 27 00:25:13.156839061 (2h33m28.937772446s) [noise=true , address=45.91.116.8:4048 , piece_id=KBWTGJIF2HUAI44ZNOGFICQMX3LQB3TDNMEIMEUZLKPCC26KBVKQ, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.156878026 (2h33m28.937811413s) [noise=true , address=77.90.31.17:4772 , piece_id=4N2LFDTN7S2YMUYO5L3RPZM7G2WXNM7FZWHEYARKUUEJDPUALQRA, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.157292455 (2h33m28.938225837s) [noise=true , address=77.90.31.10:4518 , piece_id=FDKCHHU7HLMMD72YR6FBQL3SNY3KWC5ROF5C3L4CHYVXMWI52ICQ, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.157494201 (2h33m28.938427581s) [noise=true , address=77.90.31.16:4733 , piece_id=CHIV4NLYUQSUYDARJQBR5O7VNMZMACAPP6CXLSMS6E3GWADVPHMQ, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.157651501 (2h33m28.938584887s) [noise=true , address=45.91.116.10:4081 , piece_id=YJZ54OWAQDBBKU7PCKLO5JMS6P3WIVMJM3G6GW4726MHIIGQPLZQ, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.158080485 (2h33m28.939013869s) [noise=true , address=45.91.116.6:4973 , piece_id=PEBPYGY7NDBWIXKTO6RBZJ2JMXDOSHRN7PZSOCQDJMYFGL4MQPKQ, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.158127600 (2h33m28.939060987s) [noise=true , address=77.90.31.21:4878 , piece_id=XR4HBBDT4QA44YMR5XCE3DBNZXUKMTMN2BLVX3MS6P2SFASEUL3Q, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.158413346 (2h33m28.939346724s) [noise=true , address=77.90.31.15:4699 , piece_id=4AFXJPKKDAXBWYVP2UOUYEMXTKINT6TVE7RU3ASWQBDFUHIZEOCA, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.158633735 (2h33m28.939567118s) [noise=true , address=77.90.31.14:4663 , piece_id=BZDQHDA2WG7I7KZPD3VB4PSGLRYTLKX5JPXJK3TWVMR6U4A7TCPQ, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.158790022 (2h33m28.939723394s) [noise=true , address=77.90.31.18:4807 , piece_id=VOLMX5A4ZPF6WBTMC4GOQFJCUVTB27CH3A5KYRPGUHUWGSRUIHAA, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.159238365 (2h33m28.940171753s) [noise=true , address=45.91.116.13:4153 , piece_id=A4YC7563QYYSR664U2SG5472GSECXWTZTYIBBZHIR22N35AG7U4Q, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.160717151 (2h33m28.941650518s) [noise=true , address=45.91.116.14:4191 , piece_id=HEQI2NLXJ462RJYGOHXPEUFVFXUIMLMWKRI5MKELR7PZHATNKH4A, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.161474674 (2h33m28.942408056s) [noise=true , address=104.36.82.9:4007 , piece_id=YZTTC6XE6LDSWOB5CTTZWRR35UCYOLLBGIXOHYBJVG7G5EZ5EM7A, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.162300982 (2h33m28.943234369s) [noise=true , address=45.91.116.4:4937 , piece_id=MIRJTIN2KP5QR5UPO7QT5DQBMFQY7YO56MVTXDHS54AGLWW5VQHA, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.163746259 (2h33m28.94467965s) [noise=true , address=45.91.116.8:4047 , piece_id=AKYFH3HPWUFIGRCKPSNF5KBMUS57AFU3VSD46KXJJD3KN23PEA7Q, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.164391211 (2h33m28.945324597s) [noise=true , address=104.36.82.4:4802 , piece_id=Y4JKVVI6JBN72UV5FK3QHITG3QN4PXXX34N2NCIKFA2VHW5FX3FA, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.286660997 (2h33m29.067594379s) [upload=2358]: Next returned nil segment
Jul 27 00:25:13.286684184 (2h33m29.067617547s) [upload=2358]: Waiting for error group managing segments...
Jul 27 00:25:13.292602587 (2h33m29.073535969s) [noise=true , address=45.91.116.10:4081 , piece_id=YJZ54OWAQDBBKU7PCKLO5JMS6P3WIVMJM3G6GW4726MHIIGQPLZQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.292748204 (2h33m29.073681574s) [noise=true , address=45.91.116.4:4937 , piece_id=MIRJTIN2KP5QR5UPO7QT5DQBMFQY7YO56MVTXDHS54AGLWW5VQHA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.293326557 (2h33m29.074259939s) [noise=true , address=45.91.116.8:4047 , piece_id=AKYFH3HPWUFIGRCKPSNF5KBMUS57AFU3VSD46KXJJD3KN23PEA7Q, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.293361832 (2h33m29.074295202s) [noise=true , address=77.90.31.21:4878 , piece_id=XR4HBBDT4QA44YMR5XCE3DBNZXUKMTMN2BLVX3MS6P2SFASEUL3Q, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.294304357 (2h33m29.075237733s) [noise=true , address=77.90.31.14:4663 , piece_id=BZDQHDA2WG7I7KZPD3VB4PSGLRYTLKX5JPXJK3TWVMR6U4A7TCPQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.295589750 (2h33m29.076523117s) [noise=true , address=45.91.116.8:4048 , piece_id=KBWTGJIF2HUAI44ZNOGFICQMX3LQB3TDNMEIMEUZLKPCC26KBVKQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.295634225 (2h33m29.076567588s) [noise=true , address=77.90.31.18:4807 , piece_id=VOLMX5A4ZPF6WBTMC4GOQFJCUVTB27CH3A5KYRPGUHUWGSRUIHAA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.296581902 (2h33m29.077515282s) [noise=true , address=45.91.116.13:4153 , piece_id=A4YC7563QYYSR664U2SG5472GSECXWTZTYIBBZHIR22N35AG7U4Q, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.297185145 (2h33m29.078118508s) [noise=true , address=45.91.116.6:4973 , piece_id=PEBPYGY7NDBWIXKTO6RBZJ2JMXDOSHRN7PZSOCQDJMYFGL4MQPKQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.325234980 (2h33m29.106168359s) [noise=true , address=77.90.31.15:4699 , piece_id=4AFXJPKKDAXBWYVP2UOUYEMXTKINT6TVE7RU3ASWQBDFUHIZEOCA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.360841866 (2h33m29.141775245s) [noise=true , address=45.91.116.14:4191 , piece_id=HEQI2NLXJ462RJYGOHXPEUFVFXUIMLMWKRI5MKELR7PZHATNKH4A, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:43.100924665 (2h33m58.88185803s) [noise=true , address=77.90.31.17:4772 , piece_id=4N2LFDTN7S2YMUYO5L3RPZM7G2WXNM7FZWHEYARKUUEJDPUALQRA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 159.223.143.39:56852->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:25:43.101183051 (2h33m58.882116415s) [noise=true , address=104.36.82.4:4802 , piece_id=Y4JKVVI6JBN72UV5FK3QHITG3QN4PXXX34N2NCIKFA2VHW5FX3FA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: context deadline exceeded; context deadline exceeded
Jul 27 00:25:43.101557492 (2h33m58.882490858s) [noise=true , address=77.90.31.10:4518 , piece_id=FDKCHHU7HLMMD72YR6FBQL3SNY3KWC5ROF5C3L4CHYVXMWI52ICQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:25:43.101690637 (2h33m58.882624007s) [noise=true , address=77.90.31.16:4733 , piece_id=CHIV4NLYUQSUYDARJQBR5O7VNMZMACAPP6CXLSMS6E3GWADVPHMQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded
Jul 27 00:25:43.101784731 (2h33m58.882718095s) [noise=true , address=104.36.82.9:4007 , piece_id=YZTTC6XE6LDSWOB5CTTZWRR35UCYOLLBGIXOHYBJVG7G5EZ5EM7A, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 159.223.143.39:54568->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:25:43.101803464 (2h33m58.882736842s) [seg_pos={0 0}, upload=2358]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 27 00:25:43.126349574 (2h33m58.90728296s) [upload=2358]: Done uploading segments...
Jul 27 00:25:43.126382341 (2h33m58.907315726s) [upload=2358]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.569917289s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [...........X..X......X.XX....X...X.................XX....X..] duration:
2.11329554s
FAIL (2025-07-27 00:27:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.113316557s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.027434565s
FAIL (2025-07-27 00:27:26 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
414764 (2h34m42.091348144s) [upload=2377]: Got next segment. Inline: false
Jul 27 00:26:26.310484559 (2h34m42.091417943s) [upload=2377]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 27 00:26:26.310490187 (2h34m42.091423566s) [upload=2377]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 27 00:26:26.343442614 (2h34m42.124375995s) [seg_pos={0 0}, upload=2377]: Begin upload segment...
Jul 27 00:26:26.343522729 (2h34m42.124456115s) [seg_pos={0 0}, upload=2377]: Done begin upload segment.
Jul 27 00:26:26.343575433 (2h34m42.124508821s) [noise=true , address=77.90.31.19:4875 , piece_id=D7IVBHHMQJKTMRRZWDUOQ6F3DQJ77TMOUC6ZKJOUBRBTRMFWRVGQ, seg_pos={0 0}, upload=2377]: Uploading piece...
Jul 27 00:26:26.344142525 (2h34m42.12507592s) [noise=true , address=77.90.31.16:4746 , piece_id=UDRB4TAYY5Z4PO7ZQFCPS5S7VX3LRQPU4Q5MPFB5C2F5LDI5JWXQ, seg_pos={0 0}, upload=2377]: Uploading piece...
Jul 27 00:26:26.344318450 (2h34m42.125251832s) [noise=true , address=79.127.205.246:4262 , piece_id=VZNLX55JACROS54O7JQWGBVQCYLRRS2NIBUCTVLL37GGENXO7CHQ, seg_pos={0 0}, upload=2377]: Uploading piece...
Jul 27 00:26:26.344735230 (2h34m42.125668638s) [noise=true , address=79.127.219.38:4018 , piece_id=QSFCC7YOPK2LOVYHTLFQ4MQT5AYP7XKOHO57XN5FEBDI557RIGKA, seg_pos={0 0}, upload=2377]: Uploading piece...
Jul 27 00:26:26.344869765 (2h34m42.125803388s) [noise=true , address=79.127.205.235:4691 , piece_id=ZHBI2JSZWGQE7Z2DTR6KXCBGCHTCHAUCZSKVVM75D3ZWJEWMTX3Q, seg_pos={0 0}, upload=2377]: Uploading piece...
Jul 27 00:26:26.345242507 (2h34m42.126175899s) [noise=true , address=64.49.13.140:4508 , piece_id=QFVYOC5QDA5S6STIGOW3LEHJ4QDUB4NOPLBOWNEHYXPAXJYALU5A, seg_pos={0 0}, upload=2377]: Uploading piece...
Jul 27 00:26:26.345448259 (2h34m42.126381635s) [noise=true , address=77.90.31.17:4793 , piece_id=YPY6GBN42ZLEA5GNKVT774ZY5WZTHLQZHPUJNBLHTCJK2FNHMG4A, seg_pos={0 0}, upload=2377]: Uploading piece...
Jul 27 00:26:26.345637254 (2h34m42.126570636s) [noise=true , address=77.90.31.18:4832 , piece_id=RDI5Y2IFQV5N7E454J6RC7IRA3UH5BUJLOTUH4KW22PMQ5N6FKAQ, seg_pos={0 0}, upload=2377]: Uploading piece...
Jul 27 00:26:26.345949417 (2h34m42.126882794s) [noise=true , address=64.49.13.137:4502 , piece_id=EKYOVB3VGNZEI33TUBBOEETIYNWOXX77HBKRAUQDC3TWU3CPKDYA, seg_pos={0 0}, upload=2377]: Uploading piece...
Jul 27 00:26:26.346135049 (2h34m42.127068445s) [noise=true , address=79.127.205.234:4608 , piece_id=W5HB4IJEJKCVH4H2TSDGOUSQMELEYJI4JNSBJG2C6H65K36OQOXA, seg_pos={0 0}, upload=2377]: Uploading piece...
Jul 27 00:26:26.346614775 (2h34m42.127548155s) [noise=true , address=104.36.82.46:4484 , piece_id=REF4XO4UYKOELCCQNRHPRZEBWDYSEZMNU2NMJFLWIDV3LRP6YNKA, seg_pos={0 0}, upload=2377]: Uploading piece...
Jul 27 00:26:26.347686402 (2h34m42.128619788s) [noise=true , address=104.36.82.52:4109 , piece_id=25MGIJGD3RJMLRNGKIEDS763YLSPR2FF5O5CSXWG2SE6AJL4UYIA, seg_pos={0 0}, upload=2377]: Uploading piece...
Jul 27 00:26:26.348021533 (2h34m42.128954919s) [noise=true , address=79.127.201.215:4360 , piece_id=45AIUENJKBJKZPLJOEMNO2OJVJYE4QHGQDGISWYU63PQSL3HJY2A, seg_pos={0 0}, upload=2377]: Uploading piece...
Jul 27 00:26:26.348172428 (2h34m42.129105817s) [noise=true , address=79.127.205.233:4656 , piece_id=CDINO2ITVWDVXNJ54UYBZME53OJB5R55YA2TYMA7Q5OU77KPUY5Q, seg_pos={0 0}, upload=2377]: Uploading piece...
Jul 27 00:26:26.348400972 (2h34m42.129334349s) [noise=true , address=77.90.31.21:4912 , piece_id=MVSDR6CZBQF2T7NEHXZJ23R6NZ2XOBX4T7WZZP76V5E4V2DZ5AAQ, seg_pos={0 0}, upload=2377]: Uploading piece...
Jul 27 00:26:26.348777975 (2h34m42.129711359s) [noise=true , address=104.36.82.47:4525 , piece_id=FQALJT36Z7N3V76SRSJWS3WZ3ZIP4736X76SJHHYMMWT2AULLG2A, seg_pos={0 0}, upload=2377]: Uploading piece...
Jul 27 00:26:26.348785465 (2h34m42.129718847s) [noise=true , address=45.91.116.9:4277 , piece_id=OHYZ6YEBAXIG4QQ5NPDAMFSMVMUG2AZUQ4HTM5XZVSHDKKKB5OZQ, seg_pos={0 0}, upload=2377]: Uploading piece...
Jul 27 00:26:26.349167556 (2h34m42.130100946s) [noise=true , address=79.127.220.101:4041 , piece_id=DJ4OC2BOSDKKYSHTDV7CPHPYEATJCFYBRNK5O47TRXI2SWOA3YYQ, seg_pos={0 0}, upload=2377]: Uploading piece...
Jul 27 00:26:26.439333868 (2h34m42.220267235s) [upload=2377]: Next returned nil segment
Jul 27 00:26:26.439357443 (2h34m42.220290827s) [upload=2377]: Waiting for error group managing segments...
Jul 27 00:26:26.447634820 (2h34m42.228568189s) [noise=true , address=79.127.219.38:4018 , piece_id=QSFCC7YOPK2LOVYHTLFQ4MQT5AYP7XKOHO57XN5FEBDI557RIGKA, seg_pos={0 0}, upload=2377]: Done uploading piece. err: <nil>
Jul 27 00:26:26.500993895 (2h34m42.281927258s) [noise=true , address=77.90.31.18:4832 , piece_id=RDI5Y2IFQV5N7E454J6RC7IRA3UH5BUJLOTUH4KW22PMQ5N6FKAQ, seg_pos={0 0}, upload=2377]: Done uploading piece. err: <nil>
Jul 27 00:26:26.560868998 (2h34m42.341802397s) [noise=true , address=77.90.31.21:4912 , piece_id=MVSDR6CZBQF2T7NEHXZJ23R6NZ2XOBX4T7WZZP76V5E4V2DZ5AAQ, seg_pos={0 0}, upload=2377]: Done uploading piece. err: <nil>
Jul 27 00:26:26.974790596 (2h34m42.755723974s) [noise=true , address=79.127.205.235:4691 , piece_id=ZHBI2JSZWGQE7Z2DTR6KXCBGCHTCHAUCZSKVVM75D3ZWJEWMTX3Q, seg_pos={0 0}, upload=2377]: Done uploading piece. err: <nil>
Jul 27 00:26:26.976522761 (2h34m42.757456129s) [noise=true , address=79.127.205.234:4608 , piece_id=W5HB4IJEJKCVH4H2TSDGOUSQMELEYJI4JNSBJG2C6H65K36OQOXA, seg_pos={0 0}, upload=2377]: Done uploading piece. err: <nil>
Jul 27 00:26:27.005071467 (2h34m42.786004845s) [noise=true , address=79.127.205.246:4262 , piece_id=VZNLX55JACROS54O7JQWGBVQCYLRRS2NIBUCTVLL37GGENXO7CHQ, seg_pos={0 0}, upload=2377]: Done uploading piece. err: <nil>
Jul 27 00:26:27.009050972 (2h34m42.78998441s) [noise=true , address=79.127.205.233:4656 , piece_id=CDINO2ITVWDVXNJ54UYBZME53OJB5R55YA2TYMA7Q5OU77KPUY5Q, seg_pos={0 0}, upload=2377]: Done uploading piece. err: <nil>
Jul 27 00:26:27.823461927 (2h34m43.604395297s) [noise=true , address=64.49.13.140:4508 , piece_id=QFVYOC5QDA5S6STIGOW3LEHJ4QDUB4NOPLBOWNEHYXPAXJYALU5A, seg_pos={0 0}, upload=2377]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000
Jul 27 00:26:30.225465377 (2h34m46.006398763s) [noise=true , address=79.127.220.101:4041 , piece_id=DJ4OC2BOSDKKYSHTDV7CPHPYEATJCFYBRNK5O47TRXI2SWOA3YYQ, seg_pos={0 0}, upload=2377]: Done uploading piece. err: <nil>
Jul 27 00:26:32.566199819 (2h34m48.347133205s) [noise=true , address=45.91.116.9:4277 , piece_id=OHYZ6YEBAXIG4QQ5NPDAMFSMVMUG2AZUQ4HTM5XZVSHDKKKB5OZQ, seg_pos={0 0}, upload=2377]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000
Jul 27 00:26:35.104592988 (2h34m50.885526355s) [noise=true , address=77.90.31.19:4875 , piece_id=D7IVBHHMQJKTMRRZWDUOQ6F3DQJ77TMOUC6ZKJOUBRBTRMFWRVGQ, seg_pos={0 0}, upload=2377]: Done uploading piece. err: <nil>
Jul 27 00:26:38.636390773 (2h34m54.41732415s) [noise=true , address=64.49.13.137:4502 , piece_id=EKYOVB3VGNZEI33TUBBOEETIYNWOXX77HBKRAUQDC3TWU3CPKDYA, seg_pos={0 0}, upload=2377]: Done uploading piece. err: <nil>
Jul 27 00:26:47.765018653 (2h35m3.545952029s) [noise=true , address=79.127.201.215:4360 , piece_id=45AIUENJKBJKZPLJOEMNO2OJVJYE4QHGQDGISWYU63PQSL3HJY2A, seg_pos={0 0}, upload=2377]: Done uploading piece. err: <nil>
Jul 27 00:27:26.279502407 (2h35m42.060435784s) [noise=true , address=77.90.31.17:4793 , piece_id=YPY6GBN42ZLEA5GNKVT774ZY5WZTHLQZHPUJNBLHTCJK2FNHMG4A, seg_pos={0 0}, upload=2377]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: rpc: dial tcp 77.90.31.17:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:27:26.279597697 (2h35m42.060531068s) [noise=true , address=104.36.82.46:4484 , piece_id=REF4XO4UYKOELCCQNRHPRZEBWDYSEZMNU2NMJFLWIDV3LRP6YNKA, seg_pos={0 0}, upload=2377]: Done uploading piece. err: ecclient: upload failed (node:1a5S5Ar7jNbK9MESghbWfmEgBPNSMN2tEnZeeG1kKzeX2dnSX3, address:104.36.82.46:4484): protocol: write tcp 159.223.143.39:39416->104.36.82.46:4484: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:27:26.279610783 (2h35m42.060544149s) [noise=true , address=77.90.31.16:4746 , piece_id=UDRB4TAYY5Z4PO7ZQFCPS5S7VX3LRQPU4Q5MPFB5C2F5LDI5JWXQ, seg_pos={0 0}, upload=2377]: Done uploading piece. err: ecclient: upload failed (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o, address:77.90.31.16:4746): protocol: rpc: dial tcp 77.90.31.16:4746: i/o timeout; conn closed: op
Jul 27 00:27:26.279653272 (2h35m42.060586656s) [noise=true , address=104.36.82.52:4109 , piece_id=25MGIJGD3RJMLRNGKIEDS763YLSPR2FF5O5CSXWG2SE6AJL4UYIA, seg_pos={0 0}, upload=2377]: Done uploading piece. err: ecclient: upload failed (node:1hBqAk4HrRmUKRw3RS7xUHZ4rdUMiF25jKWF7jqNrffZG8pEcj, address:104.36.82.52:4109): protocol: context deadline exceeded; context deadline exceeded
Jul 27 00:27:26.279734462 (2h35m42.060667843s) [noise=true , address=104.36.82.47:4525 , piece_id=FQALJT36Z7N3V76SRSJWS3WZ3ZIP4736X76SJHHYMMWT2AULLG2A, seg_pos={0 0}, upload=2377]: Done uploading piece. err: ecclient: upload failed (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX, address:104.36.82.47:4525): protocol: rpc: dial tcp 104.36.82.47:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:27:26.279759917 (2h35m42.060693295s) [seg_pos={0 0}, upload=2377]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 27 00:27:26.303763249 (2h35m42.084696614s) [upload=2377]: Done uploading segments...
Jul 27 00:27:26.303787289 (2h35m42.084720658s) [upload=2377]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 1m0.027458929s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
39 out of 60 is failed (threshold=3) [XXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XXX] duration:
30.031542365s
FAIL (2025-07-27 00:27:01 UTC)
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.031569043s
core
uplink
us1
vivint
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [............X..X......X.XX....X...X.................XX....X.] duration:
15.083355126s
FAIL (2025-07-27 00:26:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 15.083398316s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [.......X.X........X...XX...............................X....] duration:
40.449272833s
FAIL (2025-07-27 00:26:42 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 40.449300161s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
429.391017ms
FAIL (2025-07-27 00:26:39 UTC)
orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K
Duration: 429.504367ms
core
uplink
us1
vivint
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [.............X..X......X.XX....X...X.................XX....X] duration:
30.001614747s
FAIL (2025-07-27 00:26:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001628425s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.03427279s
FAIL (2025-07-27 00:26:26 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
tem_SegmentBegin
Jul 27 00:25:26.244624313 (2h33m42.025557709s) [seg_pos={0 0}, upload=2362]: Begin upload segment...
Jul 27 00:25:26.244702268 (2h33m42.025635657s) [seg_pos={0 0}, upload=2362]: Done begin upload segment.
Jul 27 00:25:26.244742411 (2h33m42.025675798s) [noise=true , address=79.127.201.215:4359 , piece_id=JRUQSYBOAKAA27IJ75A7WQAZABXMCJPDX2WW7B5GOPX54XWFL3WA, seg_pos={0 0}, upload=2362]: Uploading piece...
Jul 27 00:25:26.244800436 (2h33m42.025733816s) [noise=true , address=79.127.220.97:4361 , piece_id=SJB2HOKWSD7XSRM5UZIWSF4QOQUIHZP2MNHSXCWEQCBQT6OXHC6A, seg_pos={0 0}, upload=2362]: Uploading piece...
Jul 27 00:25:26.245202875 (2h33m42.026136251s) [noise=true , address=79.127.205.235:4680 , piece_id=SEWXMVEGRT3HVPTFC4K4I7J6QZTK4PX7OKTYSVOETYPYFOBOC4EA, seg_pos={0 0}, upload=2362]: Uploading piece...
Jul 27 00:25:26.245529665 (2h33m42.026463042s) [noise=true , address=77.90.31.19:4871 , piece_id=36JIQ7Q5PSOIQPSS4GKILZAAZK5UE7W4R7NBY5ZT7F3K2ACSCFRA, seg_pos={0 0}, upload=2362]: Uploading piece...
Jul 27 00:25:26.245608218 (2h33m42.0265416s) [noise=true , address=104.36.82.47:4543 , piece_id=JTI7Z26QAATCQL5HDHZFIUCV22Y7IWYDYTXCDM6YVTN36ZAEDGGQ, seg_pos={0 0}, upload=2362]: Uploading piece...
Jul 27 00:25:26.245899052 (2h33m42.026832435s) [noise=true , address=79.127.205.245:4226 , piece_id=VFIB4IMENPXPMUVNMET7RB37OCN5CCHBLZNQNOUVWBN6DPP34XEQ, seg_pos={0 0}, upload=2362]: Uploading piece...
Jul 27 00:25:26.246176303 (2h33m42.027109689s) [noise=true , address=79.127.219.38:4007 , piece_id=JAOGEBRBWPV7NLELH6NFRZFKC2F7JFTJYTKVW22XEAMOBZFAURNQ, seg_pos={0 0}, upload=2362]: Uploading piece...
Jul 27 00:25:26.246746289 (2h33m42.027679677s) [noise=true , address=79.127.220.98:4422 , piece_id=AGMXWOLUJTM5EBVR4ZP47HWQ76TIQNFRXTYI2RTNJJTOZKZT3YAQ, seg_pos={0 0}, upload=2362]: Uploading piece...
Jul 27 00:25:26.247232536 (2h33m42.028165924s) [noise=true , address=77.90.31.10:4525 , piece_id=JE6O343PIY6ED73P3HIYT7WKIXNDSJ5757GT76JYCB36YCFOJZ4A, seg_pos={0 0}, upload=2362]: Uploading piece...
Jul 27 00:25:26.247682598 (2h33m42.028615984s) [noise=true , address=64.49.13.138:4488 , piece_id=VGFVGLHCM3TEQ3YVSLCIR3NJH6P3RUT3MVXZUHC42EZA7VONKKPQ, seg_pos={0 0}, upload=2362]: Uploading piece...
Jul 27 00:25:26.248106844 (2h33m42.029040232s) [noise=true , address=79.127.205.246:4285 , piece_id=PBA675LKV7GBXPBRLOXKOHBUWLATBOVCLQNNBCXAGBM66RV32JNA, seg_pos={0 0}, upload=2362]: Uploading piece...
Jul 27 00:25:26.248905182 (2h33m42.029838567s) [noise=true , address=104.36.82.55:4223 , piece_id=XBRXK22J6BG7GEBZEOPL44A2OASRFX5NEVLOTJXTTBP2HQSVGG7Q, seg_pos={0 0}, upload=2362]: Uploading piece...
Jul 27 00:25:26.249296727 (2h33m42.030230108s) [noise=true , address=77.90.31.12:4599 , piece_id=NNL6O4W7IA6EK2GPK4FENKNLE4NGCH3CL7OCO6NGDI642QQLNEHA, seg_pos={0 0}, upload=2362]: Uploading piece...
Jul 27 00:25:26.250505895 (2h33m42.031439283s) [noise=true , address=77.90.31.15:4730 , piece_id=R3FN2M5SVCRB56DNTJVLFYZXQ6VRITBMT6ZN4TSJRZVDJ63VYYTA, seg_pos={0 0}, upload=2362]: Uploading piece...
Jul 27 00:25:26.251100881 (2h33m42.03203427s) [noise=true , address=104.36.82.3:4782 , piece_id=6CZYKHGWCWM6XYO36F6OIUUTJDQO5XLQ7RBVJEOF2MWRX7WVBVGA, seg_pos={0 0}, upload=2362]: Uploading piece...
Jul 27 00:25:26.251578633 (2h33m42.032512018s) [noise=true , address=77.90.31.13:4629 , piece_id=UMN4LM75OD64IKS66CGI2VQQ2YYF7QAF336JMGHGHEK74HZJG63A, seg_pos={0 0}, upload=2362]: Uploading piece...
Jul 27 00:25:26.252157696 (2h33m42.033091108s) [noise=true , address=77.90.31.16:4765 , piece_id=UWU3X4UEF6UZBCMZJINY2V3JIY2GYQVXHYNHFOTHYQNBPLSHGKEQ, seg_pos={0 0}, upload=2362]: Uploading piece...
Jul 27 00:25:26.252514978 (2h33m42.03344836s) [noise=true , address=104.36.82.52:4091 , piece_id=CJO2M5QTH73G5AK3P72JE5SBJWLGEMQZAJ4C7CAVH2TAG2F56ODQ, seg_pos={0 0}, upload=2362]: Uploading piece...
Jul 27 00:25:26.291604267 (2h33m42.072537643s) [noise=true , address=77.90.31.12:4599 , piece_id=NNL6O4W7IA6EK2GPK4FENKNLE4NGCH3CL7OCO6NGDI642QQLNEHA, seg_pos={0 0}, upload=2362]: Done uploading piece. err: ecclient: upload failed (node:124cEnKBraQScM26K7Zc8Z3cqTvrD9fPyWzWYutYryfXqmXQ3rn, address:77.90.31.12:4599): protocol: storage node overloaded, request limit: 2000
Jul 27 00:25:26.329006714 (2h33m42.109940097s) [upload=2362]: Next returned nil segment
Jul 27 00:25:26.329028392 (2h33m42.109961773s) [upload=2362]: Waiting for error group managing segments...
Jul 27 00:25:26.353992946 (2h33m42.134926333s) [noise=true , address=79.127.219.38:4007 , piece_id=JAOGEBRBWPV7NLELH6NFRZFKC2F7JFTJYTKVW22XEAMOBZFAURNQ, seg_pos={0 0}, upload=2362]: Done uploading piece. err: <nil>
Jul 27 00:25:26.367473401 (2h33m42.148406783s) [noise=true , address=77.90.31.15:4730 , piece_id=R3FN2M5SVCRB56DNTJVLFYZXQ6VRITBMT6ZN4TSJRZVDJ63VYYTA, seg_pos={0 0}, upload=2362]: Done uploading piece. err: <nil>
Jul 27 00:25:26.837312215 (2h33m42.618245598s) [noise=true , address=79.127.220.97:4361 , piece_id=SJB2HOKWSD7XSRM5UZIWSF4QOQUIHZP2MNHSXCWEQCBQT6OXHC6A, seg_pos={0 0}, upload=2362]: Done uploading piece. err: <nil>
Jul 27 00:25:26.853999386 (2h33m42.634932748s) [noise=true , address=79.127.205.246:4285 , piece_id=PBA675LKV7GBXPBRLOXKOHBUWLATBOVCLQNNBCXAGBM66RV32JNA, seg_pos={0 0}, upload=2362]: Done uploading piece. err: <nil>
Jul 27 00:25:26.872098702 (2h33m42.653032071s) [noise=true , address=79.127.205.245:4226 , piece_id=VFIB4IMENPXPMUVNMET7RB37OCN5CCHBLZNQNOUVWBN6DPP34XEQ, seg_pos={0 0}, upload=2362]: Done uploading piece. err: <nil>
Jul 27 00:25:27.036596515 (2h33m42.817529893s) [noise=true , address=79.127.205.235:4680 , piece_id=SEWXMVEGRT3HVPTFC4K4I7J6QZTK4PX7OKTYSVOETYPYFOBOC4EA, seg_pos={0 0}, upload=2362]: Done uploading piece. err: <nil>
Jul 27 00:25:29.524653177 (2h33m45.305586562s) [noise=true , address=79.127.220.98:4422 , piece_id=AGMXWOLUJTM5EBVR4ZP47HWQ76TIQNFRXTYI2RTNJJTOZKZT3YAQ, seg_pos={0 0}, upload=2362]: Done uploading piece. err: <nil>
Jul 27 00:25:29.826180626 (2h33m45.607113999s) [noise=true , address=77.90.31.19:4871 , piece_id=36JIQ7Q5PSOIQPSS4GKILZAAZK5UE7W4R7NBY5ZT7F3K2ACSCFRA, seg_pos={0 0}, upload=2362]: Done uploading piece. err: <nil>
Jul 27 00:25:35.766937900 (2h33m51.547871263s) [noise=true , address=64.49.13.138:4488 , piece_id=VGFVGLHCM3TEQ3YVSLCIR3NJH6P3RUT3MVXZUHC42EZA7VONKKPQ, seg_pos={0 0}, upload=2362]: Done uploading piece. err: <nil>
Jul 27 00:25:56.122288933 (2h34m11.903222302s) [noise=true , address=79.127.201.215:4359 , piece_id=JRUQSYBOAKAA27IJ75A7WQAZABXMCJPDX2WW7B5GOPX54XWFL3WA, seg_pos={0 0}, upload=2362]: Done uploading piece. err: <nil>
Jul 27 00:26:06.135670734 (2h34m21.916604104s) [noise=true , address=77.90.31.13:4629 , piece_id=UMN4LM75OD64IKS66CGI2VQQ2YYF7QAF336JMGHGHEK74HZJG63A, seg_pos={0 0}, upload=2362]: Done uploading piece. err: <nil>
Jul 27 00:26:26.168818382 (2h34m41.949751759s) [noise=true , address=104.36.82.52:4091 , piece_id=CJO2M5QTH73G5AK3P72JE5SBJWLGEMQZAJ4C7CAVH2TAG2F56ODQ, seg_pos={0 0}, upload=2362]: Done uploading piece. err: ecclient: upload failed (node:17wSQzrFc2hUimSsNwDxCgDDi4TpHkbCEH38fGKf9B8WqWzxpW, address:104.36.82.52:4091): protocol: rpc: dial tcp 104.36.82.52:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:26:26.168898636 (2h34m41.949832015s) [noise=true , address=104.36.82.47:4543 , piece_id=JTI7Z26QAATCQL5HDHZFIUCV22Y7IWYDYTXCDM6YVTN36ZAEDGGQ, seg_pos={0 0}, upload=2362]: Done uploading piece. err: ecclient: upload failed (node:12KxnBTVxhbwA79Gd8V7Mh6bWybtqttqgo5agHoHewXHAey5y7D, address:104.36.82.47:4543): protocol: write tcp 159.223.143.39:47730->104.36.82.47:4543: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:26:26.169077250 (2h34m41.950010628s) [noise=true , address=104.36.82.55:4223 , piece_id=XBRXK22J6BG7GEBZEOPL44A2OASRFX5NEVLOTJXTTBP2HQSVGG7Q, seg_pos={0 0}, upload=2362]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:104.36.82.55:4223): protocol: write tcp 159.223.143.39:38228->104.36.82.55:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:26:26.169121767 (2h34m41.950055133s) [noise=true , address=104.36.82.3:4782 , piece_id=6CZYKHGWCWM6XYO36F6OIUUTJDQO5XLQ7RBVJEOF2MWRX7WVBVGA, seg_pos={0 0}, upload=2362]: Done uploading piece. err: ecclient: upload failed (node:17C7Wx6yE15jDtqqsQwWVj444YkvHBUWkkXjoCxfXPiErwyGKQ, address:104.36.82.3:4782): protocol: write tcp 159.223.143.39:54194->104.36.82.3:4782: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:26:26.169179051 (2h34m41.950112426s) [noise=true , address=77.90.31.10:4525 , piece_id=JE6O343PIY6ED73P3HIYT7WKIXNDSJ5757GT76JYCB36YCFOJZ4A, seg_pos={0 0}, upload=2362]: Done uploading piece. err: ecclient: upload failed (node:127uTYN8tB4A6G4jW2mFpBfp7p2ziDAMYnwJLNUHv4D7f15wZ5r, address:77.90.31.10:4525): protocol: rpc: dial tcp 77.90.31.10:4525: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:26:26.169184440 (2h34m41.95011782s) [noise=true , address=77.90.31.16:4765 , piece_id=UWU3X4UEF6UZBCMZJINY2V3JIY2GYQVXHYNHFOTHYQNBPLSHGKEQ, seg_pos={0 0}, upload=2362]: Done uploading piece. err: ecclient: upload failed (node:12CSrtv2fMmFBx9zje9zBevpVzxqyhH7fL7VnHbBFEYFRPuwCKK, address:77.90.31.16:4765): protocol: write tcp 159.223.143.39:48810->77.90.31.16:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:26:26.169209994 (2h34m41.950143378s) [seg_pos={0 0}, upload=2362]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 27 00:26:26.200843887 (2h34m41.981777265s) [upload=2362]: Done uploading segments...
Jul 27 00:26:26.200875323 (2h34m41.981808701s) [upload=2362]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 1m0.034294346s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
181.275359ms
FAIL (2025-07-27 00:26:20 UTC)
orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K
Duration: 181.311348ms
core
uplink
us1
vivint
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
42 out of 60 is failed (threshold=3) [XXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX.] duration:
2.489898626s
FAIL (2025-07-27 00:26:15 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
5.226875599 (2h33m1.007808984s) [noise=true , address=104.36.82.4:4801 , piece_id=AARTJBSLFRFJNTY64JVS7C4I5WOW3PMUDP3GSCJ4WDDOMZBHJQUQ, seg_pos={0 0}, upload=2341]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 159.223.143.39:35016->104.36.82.4:4801: use of closed network connection; context deadline exceeded
Jul 27 00:24:45.226924742 (2h33m1.007858123s) [seg_pos={0 0}, upload=2341]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 27 00:24:45.260403710 (2h33m1.041337086s) [upload=2341]: Done uploading segments...
Jul 27 00:24:45.260436570 (2h33m1.041369954s) [upload=2341]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Jul 27 00:25:13.130330845 (2h33m28.911264233s) [upload=2358]: Starting upload
Jul 27 00:25:13.130427709 (2h33m28.911361097s) [upload=2358]: Done starting upload
Jul 27 00:25:13.130504194 (2h33m28.911437572s) [upload=2358]: Uploading segments...
Jul 27 00:25:13.130768741 (2h33m28.911702127s) [upload=2358]: Got next segment. Inline: false
Jul 27 00:25:13.130825048 (2h33m28.911758432s) [upload=2358]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 27 00:25:13.130832706 (2h33m28.911766093s) [upload=2358]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 27 00:25:13.156703675 (2h33m28.937637054s) [seg_pos={0 0}, upload=2358]: Begin upload segment...
Jul 27 00:25:13.156790074 (2h33m28.937723451s) [seg_pos={0 0}, upload=2358]: Done begin upload segment.
Jul 27 00:25:13.156839061 (2h33m28.937772446s) [noise=true , address=45.91.116.8:4048 , piece_id=KBWTGJIF2HUAI44ZNOGFICQMX3LQB3TDNMEIMEUZLKPCC26KBVKQ, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.156878026 (2h33m28.937811413s) [noise=true , address=77.90.31.17:4772 , piece_id=4N2LFDTN7S2YMUYO5L3RPZM7G2WXNM7FZWHEYARKUUEJDPUALQRA, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.157292455 (2h33m28.938225837s) [noise=true , address=77.90.31.10:4518 , piece_id=FDKCHHU7HLMMD72YR6FBQL3SNY3KWC5ROF5C3L4CHYVXMWI52ICQ, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.157494201 (2h33m28.938427581s) [noise=true , address=77.90.31.16:4733 , piece_id=CHIV4NLYUQSUYDARJQBR5O7VNMZMACAPP6CXLSMS6E3GWADVPHMQ, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.157651501 (2h33m28.938584887s) [noise=true , address=45.91.116.10:4081 , piece_id=YJZ54OWAQDBBKU7PCKLO5JMS6P3WIVMJM3G6GW4726MHIIGQPLZQ, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.158080485 (2h33m28.939013869s) [noise=true , address=45.91.116.6:4973 , piece_id=PEBPYGY7NDBWIXKTO6RBZJ2JMXDOSHRN7PZSOCQDJMYFGL4MQPKQ, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.158127600 (2h33m28.939060987s) [noise=true , address=77.90.31.21:4878 , piece_id=XR4HBBDT4QA44YMR5XCE3DBNZXUKMTMN2BLVX3MS6P2SFASEUL3Q, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.158413346 (2h33m28.939346724s) [noise=true , address=77.90.31.15:4699 , piece_id=4AFXJPKKDAXBWYVP2UOUYEMXTKINT6TVE7RU3ASWQBDFUHIZEOCA, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.158633735 (2h33m28.939567118s) [noise=true , address=77.90.31.14:4663 , piece_id=BZDQHDA2WG7I7KZPD3VB4PSGLRYTLKX5JPXJK3TWVMR6U4A7TCPQ, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.158790022 (2h33m28.939723394s) [noise=true , address=77.90.31.18:4807 , piece_id=VOLMX5A4ZPF6WBTMC4GOQFJCUVTB27CH3A5KYRPGUHUWGSRUIHAA, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.159238365 (2h33m28.940171753s) [noise=true , address=45.91.116.13:4153 , piece_id=A4YC7563QYYSR664U2SG5472GSECXWTZTYIBBZHIR22N35AG7U4Q, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.160717151 (2h33m28.941650518s) [noise=true , address=45.91.116.14:4191 , piece_id=HEQI2NLXJ462RJYGOHXPEUFVFXUIMLMWKRI5MKELR7PZHATNKH4A, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.161474674 (2h33m28.942408056s) [noise=true , address=104.36.82.9:4007 , piece_id=YZTTC6XE6LDSWOB5CTTZWRR35UCYOLLBGIXOHYBJVG7G5EZ5EM7A, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.162300982 (2h33m28.943234369s) [noise=true , address=45.91.116.4:4937 , piece_id=MIRJTIN2KP5QR5UPO7QT5DQBMFQY7YO56MVTXDHS54AGLWW5VQHA, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.163746259 (2h33m28.94467965s) [noise=true , address=45.91.116.8:4047 , piece_id=AKYFH3HPWUFIGRCKPSNF5KBMUS57AFU3VSD46KXJJD3KN23PEA7Q, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.164391211 (2h33m28.945324597s) [noise=true , address=104.36.82.4:4802 , piece_id=Y4JKVVI6JBN72UV5FK3QHITG3QN4PXXX34N2NCIKFA2VHW5FX3FA, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.286660997 (2h33m29.067594379s) [upload=2358]: Next returned nil segment
Jul 27 00:25:13.286684184 (2h33m29.067617547s) [upload=2358]: Waiting for error group managing segments...
Jul 27 00:25:13.292602587 (2h33m29.073535969s) [noise=true , address=45.91.116.10:4081 , piece_id=YJZ54OWAQDBBKU7PCKLO5JMS6P3WIVMJM3G6GW4726MHIIGQPLZQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.292748204 (2h33m29.073681574s) [noise=true , address=45.91.116.4:4937 , piece_id=MIRJTIN2KP5QR5UPO7QT5DQBMFQY7YO56MVTXDHS54AGLWW5VQHA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.293326557 (2h33m29.074259939s) [noise=true , address=45.91.116.8:4047 , piece_id=AKYFH3HPWUFIGRCKPSNF5KBMUS57AFU3VSD46KXJJD3KN23PEA7Q, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.293361832 (2h33m29.074295202s) [noise=true , address=77.90.31.21:4878 , piece_id=XR4HBBDT4QA44YMR5XCE3DBNZXUKMTMN2BLVX3MS6P2SFASEUL3Q, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.294304357 (2h33m29.075237733s) [noise=true , address=77.90.31.14:4663 , piece_id=BZDQHDA2WG7I7KZPD3VB4PSGLRYTLKX5JPXJK3TWVMR6U4A7TCPQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.295589750 (2h33m29.076523117s) [noise=true , address=45.91.116.8:4048 , piece_id=KBWTGJIF2HUAI44ZNOGFICQMX3LQB3TDNMEIMEUZLKPCC26KBVKQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.295634225 (2h33m29.076567588s) [noise=true , address=77.90.31.18:4807 , piece_id=VOLMX5A4ZPF6WBTMC4GOQFJCUVTB27CH3A5KYRPGUHUWGSRUIHAA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.296581902 (2h33m29.077515282s) [noise=true , address=45.91.116.13:4153 , piece_id=A4YC7563QYYSR664U2SG5472GSECXWTZTYIBBZHIR22N35AG7U4Q, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.297185145 (2h33m29.078118508s) [noise=true , address=45.91.116.6:4973 , piece_id=PEBPYGY7NDBWIXKTO6RBZJ2JMXDOSHRN7PZSOCQDJMYFGL4MQPKQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.325234980 (2h33m29.106168359s) [noise=true , address=77.90.31.15:4699 , piece_id=4AFXJPKKDAXBWYVP2UOUYEMXTKINT6TVE7RU3ASWQBDFUHIZEOCA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.360841866 (2h33m29.141775245s) [noise=true , address=45.91.116.14:4191 , piece_id=HEQI2NLXJ462RJYGOHXPEUFVFXUIMLMWKRI5MKELR7PZHATNKH4A, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:43.100924665 (2h33m58.88185803s) [noise=true , address=77.90.31.17:4772 , piece_id=4N2LFDTN7S2YMUYO5L3RPZM7G2WXNM7FZWHEYARKUUEJDPUALQRA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 159.223.143.39:56852->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:25:43.101183051 (2h33m58.882116415s) [noise=true , address=104.36.82.4:4802 , piece_id=Y4JKVVI6JBN72UV5FK3QHITG3QN4PXXX34N2NCIKFA2VHW5FX3FA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: context deadline exceeded; context deadline exceeded
Jul 27 00:25:43.101557492 (2h33m58.882490858s) [noise=true , address=77.90.31.10:4518 , piece_id=FDKCHHU7HLMMD72YR6FBQL3SNY3KWC5ROF5C3L4CHYVXMWI52ICQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:25:43.101690637 (2h33m58.882624007s) [noise=true , address=77.90.31.16:4733 , piece_id=CHIV4NLYUQSUYDARJQBR5O7VNMZMACAPP6CXLSMS6E3GWADVPHMQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded
Jul 27 00:25:43.101784731 (2h33m58.882718095s) [noise=true , address=104.36.82.9:4007 , piece_id=YZTTC6XE6LDSWOB5CTTZWRR35UCYOLLBGIXOHYBJVG7G5EZ5EM7A, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 159.223.143.39:54568->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:25:43.101803464 (2h33m58.882736842s) [seg_pos={0 0}, upload=2358]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 27 00:25:43.126349574 (2h33m58.90728296s) [upload=2358]: Done uploading segments...
Jul 27 00:25:43.126382341 (2h33m58.907315726s) [upload=2358]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.489926749s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
39 out of 60 is failed (threshold=3) [XXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XX] duration:
30.028556323s
FAIL (2025-07-27 00:26:14 UTC)
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 30.028648049s
core
uplink
us1
vivint
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [........X.X........X...XX...............................X...] duration:
46.265522184s
FAIL (2025-07-27 00:25:53 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 46.265558064s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [..............X..X......X.XX....X...X.................XX....] duration:
2.562520385s
FAIL (2025-07-27 00:25:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.562546563s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
43 out of 60 is failed (threshold=3) [XXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX] duration:
30.027403278s
FAIL (2025-07-27 00:25:43 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
5.226875599 (2h33m1.007808984s) [noise=true , address=104.36.82.4:4801 , piece_id=AARTJBSLFRFJNTY64JVS7C4I5WOW3PMUDP3GSCJ4WDDOMZBHJQUQ, seg_pos={0 0}, upload=2341]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 159.223.143.39:35016->104.36.82.4:4801: use of closed network connection; context deadline exceeded
Jul 27 00:24:45.226924742 (2h33m1.007858123s) [seg_pos={0 0}, upload=2341]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 27 00:24:45.260403710 (2h33m1.041337086s) [upload=2341]: Done uploading segments...
Jul 27 00:24:45.260436570 (2h33m1.041369954s) [upload=2341]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Jul 27 00:25:13.130330845 (2h33m28.911264233s) [upload=2358]: Starting upload
Jul 27 00:25:13.130427709 (2h33m28.911361097s) [upload=2358]: Done starting upload
Jul 27 00:25:13.130504194 (2h33m28.911437572s) [upload=2358]: Uploading segments...
Jul 27 00:25:13.130768741 (2h33m28.911702127s) [upload=2358]: Got next segment. Inline: false
Jul 27 00:25:13.130825048 (2h33m28.911758432s) [upload=2358]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 27 00:25:13.130832706 (2h33m28.911766093s) [upload=2358]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 27 00:25:13.156703675 (2h33m28.937637054s) [seg_pos={0 0}, upload=2358]: Begin upload segment...
Jul 27 00:25:13.156790074 (2h33m28.937723451s) [seg_pos={0 0}, upload=2358]: Done begin upload segment.
Jul 27 00:25:13.156839061 (2h33m28.937772446s) [noise=true , address=45.91.116.8:4048 , piece_id=KBWTGJIF2HUAI44ZNOGFICQMX3LQB3TDNMEIMEUZLKPCC26KBVKQ, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.156878026 (2h33m28.937811413s) [noise=true , address=77.90.31.17:4772 , piece_id=4N2LFDTN7S2YMUYO5L3RPZM7G2WXNM7FZWHEYARKUUEJDPUALQRA, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.157292455 (2h33m28.938225837s) [noise=true , address=77.90.31.10:4518 , piece_id=FDKCHHU7HLMMD72YR6FBQL3SNY3KWC5ROF5C3L4CHYVXMWI52ICQ, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.157494201 (2h33m28.938427581s) [noise=true , address=77.90.31.16:4733 , piece_id=CHIV4NLYUQSUYDARJQBR5O7VNMZMACAPP6CXLSMS6E3GWADVPHMQ, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.157651501 (2h33m28.938584887s) [noise=true , address=45.91.116.10:4081 , piece_id=YJZ54OWAQDBBKU7PCKLO5JMS6P3WIVMJM3G6GW4726MHIIGQPLZQ, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.158080485 (2h33m28.939013869s) [noise=true , address=45.91.116.6:4973 , piece_id=PEBPYGY7NDBWIXKTO6RBZJ2JMXDOSHRN7PZSOCQDJMYFGL4MQPKQ, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.158127600 (2h33m28.939060987s) [noise=true , address=77.90.31.21:4878 , piece_id=XR4HBBDT4QA44YMR5XCE3DBNZXUKMTMN2BLVX3MS6P2SFASEUL3Q, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.158413346 (2h33m28.939346724s) [noise=true , address=77.90.31.15:4699 , piece_id=4AFXJPKKDAXBWYVP2UOUYEMXTKINT6TVE7RU3ASWQBDFUHIZEOCA, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.158633735 (2h33m28.939567118s) [noise=true , address=77.90.31.14:4663 , piece_id=BZDQHDA2WG7I7KZPD3VB4PSGLRYTLKX5JPXJK3TWVMR6U4A7TCPQ, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.158790022 (2h33m28.939723394s) [noise=true , address=77.90.31.18:4807 , piece_id=VOLMX5A4ZPF6WBTMC4GOQFJCUVTB27CH3A5KYRPGUHUWGSRUIHAA, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.159238365 (2h33m28.940171753s) [noise=true , address=45.91.116.13:4153 , piece_id=A4YC7563QYYSR664U2SG5472GSECXWTZTYIBBZHIR22N35AG7U4Q, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.160717151 (2h33m28.941650518s) [noise=true , address=45.91.116.14:4191 , piece_id=HEQI2NLXJ462RJYGOHXPEUFVFXUIMLMWKRI5MKELR7PZHATNKH4A, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.161474674 (2h33m28.942408056s) [noise=true , address=104.36.82.9:4007 , piece_id=YZTTC6XE6LDSWOB5CTTZWRR35UCYOLLBGIXOHYBJVG7G5EZ5EM7A, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.162300982 (2h33m28.943234369s) [noise=true , address=45.91.116.4:4937 , piece_id=MIRJTIN2KP5QR5UPO7QT5DQBMFQY7YO56MVTXDHS54AGLWW5VQHA, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.163746259 (2h33m28.94467965s) [noise=true , address=45.91.116.8:4047 , piece_id=AKYFH3HPWUFIGRCKPSNF5KBMUS57AFU3VSD46KXJJD3KN23PEA7Q, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.164391211 (2h33m28.945324597s) [noise=true , address=104.36.82.4:4802 , piece_id=Y4JKVVI6JBN72UV5FK3QHITG3QN4PXXX34N2NCIKFA2VHW5FX3FA, seg_pos={0 0}, upload=2358]: Uploading piece...
Jul 27 00:25:13.286660997 (2h33m29.067594379s) [upload=2358]: Next returned nil segment
Jul 27 00:25:13.286684184 (2h33m29.067617547s) [upload=2358]: Waiting for error group managing segments...
Jul 27 00:25:13.292602587 (2h33m29.073535969s) [noise=true , address=45.91.116.10:4081 , piece_id=YJZ54OWAQDBBKU7PCKLO5JMS6P3WIVMJM3G6GW4726MHIIGQPLZQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.292748204 (2h33m29.073681574s) [noise=true , address=45.91.116.4:4937 , piece_id=MIRJTIN2KP5QR5UPO7QT5DQBMFQY7YO56MVTXDHS54AGLWW5VQHA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.293326557 (2h33m29.074259939s) [noise=true , address=45.91.116.8:4047 , piece_id=AKYFH3HPWUFIGRCKPSNF5KBMUS57AFU3VSD46KXJJD3KN23PEA7Q, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.293361832 (2h33m29.074295202s) [noise=true , address=77.90.31.21:4878 , piece_id=XR4HBBDT4QA44YMR5XCE3DBNZXUKMTMN2BLVX3MS6P2SFASEUL3Q, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.294304357 (2h33m29.075237733s) [noise=true , address=77.90.31.14:4663 , piece_id=BZDQHDA2WG7I7KZPD3VB4PSGLRYTLKX5JPXJK3TWVMR6U4A7TCPQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.295589750 (2h33m29.076523117s) [noise=true , address=45.91.116.8:4048 , piece_id=KBWTGJIF2HUAI44ZNOGFICQMX3LQB3TDNMEIMEUZLKPCC26KBVKQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.295634225 (2h33m29.076567588s) [noise=true , address=77.90.31.18:4807 , piece_id=VOLMX5A4ZPF6WBTMC4GOQFJCUVTB27CH3A5KYRPGUHUWGSRUIHAA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.296581902 (2h33m29.077515282s) [noise=true , address=45.91.116.13:4153 , piece_id=A4YC7563QYYSR664U2SG5472GSECXWTZTYIBBZHIR22N35AG7U4Q, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.297185145 (2h33m29.078118508s) [noise=true , address=45.91.116.6:4973 , piece_id=PEBPYGY7NDBWIXKTO6RBZJ2JMXDOSHRN7PZSOCQDJMYFGL4MQPKQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.325234980 (2h33m29.106168359s) [noise=true , address=77.90.31.15:4699 , piece_id=4AFXJPKKDAXBWYVP2UOUYEMXTKINT6TVE7RU3ASWQBDFUHIZEOCA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:13.360841866 (2h33m29.141775245s) [noise=true , address=45.91.116.14:4191 , piece_id=HEQI2NLXJ462RJYGOHXPEUFVFXUIMLMWKRI5MKELR7PZHATNKH4A, seg_pos={0 0}, upload=2358]: Done uploading piece. err: <nil>
Jul 27 00:25:43.100924665 (2h33m58.88185803s) [noise=true , address=77.90.31.17:4772 , piece_id=4N2LFDTN7S2YMUYO5L3RPZM7G2WXNM7FZWHEYARKUUEJDPUALQRA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 159.223.143.39:56852->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:25:43.101183051 (2h33m58.882116415s) [noise=true , address=104.36.82.4:4802 , piece_id=Y4JKVVI6JBN72UV5FK3QHITG3QN4PXXX34N2NCIKFA2VHW5FX3FA, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: context deadline exceeded; context deadline exceeded
Jul 27 00:25:43.101557492 (2h33m58.882490858s) [noise=true , address=77.90.31.10:4518 , piece_id=FDKCHHU7HLMMD72YR6FBQL3SNY3KWC5ROF5C3L4CHYVXMWI52ICQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:25:43.101690637 (2h33m58.882624007s) [noise=true , address=77.90.31.16:4733 , piece_id=CHIV4NLYUQSUYDARJQBR5O7VNMZMACAPP6CXLSMS6E3GWADVPHMQ, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded
Jul 27 00:25:43.101784731 (2h33m58.882718095s) [noise=true , address=104.36.82.9:4007 , piece_id=YZTTC6XE6LDSWOB5CTTZWRR35UCYOLLBGIXOHYBJVG7G5EZ5EM7A, seg_pos={0 0}, upload=2358]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 159.223.143.39:54568->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:25:43.101803464 (2h33m58.882736842s) [seg_pos={0 0}, upload=2358]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 27 00:25:43.126349574 (2h33m58.90728296s) [upload=2358]: Done uploading segments...
Jul 27 00:25:43.126382341 (2h33m58.907315726s) [upload=2358]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.02743408s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.029152973s
FAIL (2025-07-27 00:25:26 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
DOTA, seg_pos={0 0}, upload=2345]: Uploading piece...
Jul 27 00:24:26.144717031 (2h32m41.925650412s) [noise=true , address=79.127.205.233:4646 , piece_id=MVFEMCQGCRD66M2354ZI4PT6XKCGGASCZ32ETPZ2R6FRW5HP7ICA, seg_pos={0 0}, upload=2345]: Uploading piece...
Jul 27 00:24:26.145138585 (2h32m41.926071965s) [noise=true , address=104.36.82.55:4217 , piece_id=2CD4YAEUJEXVEE4DE77CNVMQ6FIO3D6KSZE4NGK44LKF4LN5PBIA, seg_pos={0 0}, upload=2345]: Uploading piece...
Jul 27 00:24:26.145255059 (2h32m41.926188438s) [noise=true , address=77.90.31.21:4882 , piece_id=IXQM5LKLCCZVTI43446VNZQMGDZOG6I27OUIFFWKN7COEIPLJYGA, seg_pos={0 0}, upload=2345]: Uploading piece...
Jul 27 00:24:26.145500697 (2h32m41.926434074s) [noise=true , address=104.36.82.46:4495 , piece_id=CINFJ5JW56RNUMXAW2GKNFQJFJZVFN3ZIYFSXY2UEBAW62KDCFHA, seg_pos={0 0}, upload=2345]: Uploading piece...
Jul 27 00:24:26.145725984 (2h32m41.926659369s) [noise=true , address=104.36.82.47:4528 , piece_id=BUNIMSFRCCNX3XF6NROPGZLNRKLLQCTIQHHTI4IMINZUJWOCMDVQ, seg_pos={0 0}, upload=2345]: Uploading piece...
Jul 27 00:24:26.145859916 (2h32m41.926793285s) [noise=true , address=79.127.220.101:4048 , piece_id=RAHM5QA5YM4YDONCLCBHTUAE2HSIBDGS3DTYIXEPLJSVYWU4W2VA, seg_pos={0 0}, upload=2345]: Uploading piece...
Jul 27 00:24:26.146193690 (2h32m41.927127078s) [noise=true , address=77.90.31.11:4573 , piece_id=TCL6SVCEE72Q5L3C4AP3P5YIDOXRFR3L7V4V5T4GB75LFUQHS52Q, seg_pos={0 0}, upload=2345]: Uploading piece...
Jul 27 00:24:26.146672841 (2h32m41.927606226s) [noise=true , address=104.36.82.54:4161 , piece_id=64PWUSYXKZ2NAWMXHOI3VSG3N3A32PMSF4H5GLLVZJSN3KH6DQSA, seg_pos={0 0}, upload=2345]: Uploading piece...
Jul 27 00:24:26.148131952 (2h32m41.92906534s) [noise=true , address=77.90.31.12:4597 , piece_id=R22R3IALNL52S27QFAHBAWYGK7AG5W2MK7ZE5O66AHWS2GAOG4QQ, seg_pos={0 0}, upload=2345]: Uploading piece...
Jul 27 00:24:26.148644203 (2h32m41.929577591s) [noise=true , address=79.127.205.245:4226 , piece_id=BGDKLU4H4UVGF23EVXUJMFKLQBFFFZWUDSYRKHNSLXYACJTHLAKA, seg_pos={0 0}, upload=2345]: Uploading piece...
Jul 27 00:24:26.149154530 (2h32m41.930087912s) [noise=true , address=79.127.205.246:4283 , piece_id=TDQ5XREXHDRRXQ2SNAEFVLSPPE2NUU3LM7GPC7OKM36HSHWIITNA, seg_pos={0 0}, upload=2345]: Uploading piece...
Jul 27 00:24:26.149164538 (2h32m41.930097909s) [noise=true , address=77.90.31.15:4723 , piece_id=2CSXXBCMY6NGHRNFSVI472IAHBWRA4AQDTWHWQMTOG5JSBSDYQDQ, seg_pos={0 0}, upload=2345]: Uploading piece...
Jul 27 00:24:26.149632311 (2h32m41.930565704s) [noise=true , address=104.36.82.53:4141 , piece_id=FTW4KKWZC3CXQEAQSPEJEZFYIR4IVR63LN5SXEITGJEBIBINNYTQ, seg_pos={0 0}, upload=2345]: Uploading piece...
Jul 27 00:24:26.150650371 (2h32m41.931583759s) [noise=true , address=79.127.201.215:4359 , piece_id=I346M37ZESQ546M46GVTY55U56W6ZG3LAAQHIQOATN64RO6KG7IA, seg_pos={0 0}, upload=2345]: Uploading piece...
Jul 27 00:24:26.151188549 (2h32m41.932121935s) [noise=true , address=77.90.31.16:4764 , piece_id=OROXSACR2VZOSI7BGUCMAYZVUSOS3PZXNCNCMBGC22XHVLI4LJNQ, seg_pos={0 0}, upload=2345]: Uploading piece...
Jul 27 00:24:26.152270751 (2h32m41.933204131s) [noise=true , address=77.90.31.18:4825 , piece_id=PZISWJGVHM2DQWW7ZA6IB7FEYCNRF44TKKZTZL2BRGLDSD2Z623A, seg_pos={0 0}, upload=2345]: Uploading piece...
Jul 27 00:24:26.152829565 (2h32m41.933762954s) [noise=true , address=79.127.219.38:4006 , piece_id=OI2SLC56PJXOMDVF7LSGGEZ5DAQ5EP32L76BCPO3OF2AQAWM3GIQ, seg_pos={0 0}, upload=2345]: Uploading piece...
Jul 27 00:24:26.288975838 (2h32m42.069909337s) [upload=2345]: Next returned nil segment
Jul 27 00:24:26.288996776 (2h32m42.069930166s) [upload=2345]: Waiting for error group managing segments...
Jul 27 00:24:26.291729785 (2h32m42.072663165s) [noise=true , address=79.127.220.101:4048 , piece_id=RAHM5QA5YM4YDONCLCBHTUAE2HSIBDGS3DTYIXEPLJSVYWU4W2VA, seg_pos={0 0}, upload=2345]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000
Jul 27 00:24:26.304405017 (2h32m42.085338401s) [noise=true , address=79.127.219.38:4006 , piece_id=OI2SLC56PJXOMDVF7LSGGEZ5DAQ5EP32L76BCPO3OF2AQAWM3GIQ, seg_pos={0 0}, upload=2345]: Done uploading piece. err: <nil>
Jul 27 00:24:26.309698318 (2h32m42.090631683s) [noise=true , address=77.90.31.15:4723 , piece_id=2CSXXBCMY6NGHRNFSVI472IAHBWRA4AQDTWHWQMTOG5JSBSDYQDQ, seg_pos={0 0}, upload=2345]: Done uploading piece. err: <nil>
Jul 27 00:24:26.340222509 (2h32m42.12115588s) [noise=true , address=77.90.31.21:4882 , piece_id=IXQM5LKLCCZVTI43446VNZQMGDZOG6I27OUIFFWKN7COEIPLJYGA, seg_pos={0 0}, upload=2345]: Done uploading piece. err: <nil>
Jul 27 00:24:26.361893801 (2h32m42.142827179s) [noise=true , address=77.90.31.18:4825 , piece_id=PZISWJGVHM2DQWW7ZA6IB7FEYCNRF44TKKZTZL2BRGLDSD2Z623A, seg_pos={0 0}, upload=2345]: Done uploading piece. err: <nil>
Jul 27 00:24:26.752522993 (2h32m42.533456358s) [noise=true , address=79.127.205.246:4283 , piece_id=TDQ5XREXHDRRXQ2SNAEFVLSPPE2NUU3LM7GPC7OKM36HSHWIITNA, seg_pos={0 0}, upload=2345]: Done uploading piece. err: <nil>
Jul 27 00:24:26.779164681 (2h32m42.560098057s) [noise=true , address=79.127.205.233:4646 , piece_id=MVFEMCQGCRD66M2354ZI4PT6XKCGGASCZ32ETPZ2R6FRW5HP7ICA, seg_pos={0 0}, upload=2345]: Done uploading piece. err: <nil>
Jul 27 00:24:26.779792648 (2h32m42.560726017s) [noise=true , address=79.127.205.245:4226 , piece_id=BGDKLU4H4UVGF23EVXUJMFKLQBFFFZWUDSYRKHNSLXYACJTHLAKA, seg_pos={0 0}, upload=2345]: Done uploading piece. err: <nil>
Jul 27 00:24:27.345648921 (2h32m43.126582282s) [noise=true , address=77.90.31.12:4597 , piece_id=R22R3IALNL52S27QFAHBAWYGK7AG5W2MK7ZE5O66AHWS2GAOG4QQ, seg_pos={0 0}, upload=2345]: Done uploading piece. err: <nil>
Jul 27 00:24:49.617145628 (2h33m5.398078998s) [noise=true , address=79.127.201.215:4359 , piece_id=I346M37ZESQ546M46GVTY55U56W6ZG3LAAQHIQOATN64RO6KG7IA, seg_pos={0 0}, upload=2345]: Done uploading piece. err: <nil>
Jul 27 00:25:26.074779638 (2h33m41.855713013s) [noise=true , address=104.36.82.55:4217 , piece_id=2CD4YAEUJEXVEE4DE77CNVMQ6FIO3D6KSZE4NGK44LKF4LN5PBIA, seg_pos={0 0}, upload=2345]: Done uploading piece. err: ecclient: upload failed (node:12bw9V8UMi7pFxqJ8fh22x5ZPmtJUJyJGw8hD9NhQcfHdfx4cZj, address:104.36.82.55:4217): protocol: rpc: dial tcp 104.36.82.55:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:25:26.074846936 (2h33m41.855780311s) [noise=true , address=104.36.82.47:4528 , piece_id=BUNIMSFRCCNX3XF6NROPGZLNRKLLQCTIQHHTI4IMINZUJWOCMDVQ, seg_pos={0 0}, upload=2345]: Done uploading piece. err: ecclient: upload failed (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT, address:104.36.82.47:4528): protocol: rpc: dial tcp 104.36.82.47:4528: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:25:26.074924665 (2h33m41.855858027s) [noise=true , address=77.90.31.11:4573 , piece_id=TCL6SVCEE72Q5L3C4AP3P5YIDOXRFR3L7V4V5T4GB75LFUQHS52Q, seg_pos={0 0}, upload=2345]: Done uploading piece. err: ecclient: upload failed (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV, address:77.90.31.11:4573): protocol: read tcp 159.223.143.39:50386->77.90.31.11:4573: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:25:26.074968338 (2h33m41.855901715s) [noise=true , address=77.90.31.16:4764 , piece_id=OROXSACR2VZOSI7BGUCMAYZVUSOS3PZXNCNCMBGC22XHVLI4LJNQ, seg_pos={0 0}, upload=2345]: Done uploading piece. err: ecclient: upload failed (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT, address:77.90.31.16:4764): protocol: rpc: dial tcp 77.90.31.16:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:25:26.074981473 (2h33m41.85591485s) [noise=true , address=104.36.82.46:4495 , piece_id=CINFJ5JW56RNUMXAW2GKNFQJFJZVFN3ZIYFSXY2UEBAW62KDCFHA, seg_pos={0 0}, upload=2345]: Done uploading piece. err: ecclient: upload failed (node:1YpHarCJ3ouMGTs59udhU7az9h68ELEHTa2aTAax2chR2Gux1x, address:104.36.82.46:4495): protocol: rpc: dial tcp 104.36.82.46:4495: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:25:26.075130480 (2h33m41.856063857s) [noise=true , address=104.36.82.54:4161 , piece_id=64PWUSYXKZ2NAWMXHOI3VSG3N3A32PMSF4H5GLLVZJSN3KH6DQSA, seg_pos={0 0}, upload=2345]: Done uploading piece. err: ecclient: upload failed (node:12pyhMcYjL5b9posZurgKXBnCHaMhaEjL3q3YWyc3aWkcYwbvTD, address:104.36.82.54:4161): protocol: rpc: dial tcp 104.36.82.54:4161: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:25:26.075179827 (2h33m41.856113192s) [noise=true , address=104.36.82.52:4101 , piece_id=KJAT3DZY43IUD2X7VRFL5KTLTFKPYFBTH7IX2HSMBAPRFUTXDOTA, seg_pos={0 0}, upload=2345]: Done uploading piece. err: ecclient: upload failed (node:12WWXeBjFZR9PzciNfs35fFcx87N9GwDpMsPHnksFDugtaBGWp4, address:104.36.82.52:4101): protocol: write tcp 159.223.143.39:47750->104.36.82.52:4101: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:25:26.075198465 (2h33m41.856131826s) [noise=true , address=104.36.82.53:4141 , piece_id=FTW4KKWZC3CXQEAQSPEJEZFYIR4IVR63LN5SXEITGJEBIBINNYTQ, seg_pos={0 0}, upload=2345]: Done uploading piece. err: ecclient: upload failed (node:15wUF135h4QsyvF7Cr3eb6rxtx8KPz1LHbgRjQYwiF5xpmE2nn, address:104.36.82.53:4141): protocol: rpc: dial tcp 104.36.82.53:4141: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:25:26.075236070 (2h33m41.856169451s) [seg_pos={0 0}, upload=2345]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Jul 27 00:25:26.100957851 (2h33m41.881891231s) [upload=2345]: Done uploading segments...
Jul 27 00:25:26.100988962 (2h33m41.881922344s) [upload=2345]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 1m0.029205696s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
39 out of 60 is failed (threshold=3) [XXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.X] duration:
30.032190265s
FAIL (2025-07-27 00:25:18 UTC)
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.032224868s
core
uplink
us1
vivint
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [...............X..X......X.XX....X...X.................XX...] duration:
10.516559351s
FAIL (2025-07-27 00:24:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 10.516593906s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
43 out of 60 is failed (threshold=3) [XXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXX] duration:
30.037846199s
FAIL (2025-07-27 00:24:45 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
75): protocol: write tcp 159.223.143.39:53440->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:23:35.802082296 (2h31m51.583015681s) [seg_pos={0 0}, upload=2325]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Jul 27 00:23:35.830966119 (2h31m51.611899487s) [upload=2325]: Done uploading segments...
Jul 27 00:23:35.830993756 (2h31m51.611927137s) [upload=2325]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Jul 27 00:24:15.256062013 (2h32m31.036995375s) [upload=2341]: Starting upload
Jul 27 00:24:15.256158699 (2h32m31.037092079s) [upload=2341]: Done starting upload
Jul 27 00:24:15.256302903 (2h32m31.037236284s) [upload=2341]: Uploading segments...
Jul 27 00:24:15.256454993 (2h32m31.037388363s) [upload=2341]: Got next segment. Inline: false
Jul 27 00:24:15.256508209 (2h32m31.037441584s) [upload=2341]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 27 00:24:15.256513899 (2h32m31.037447274s) [upload=2341]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 27 00:24:15.290479823 (2h32m31.071413197s) [seg_pos={0 0}, upload=2341]: Begin upload segment...
Jul 27 00:24:15.290522174 (2h32m31.071455556s) [seg_pos={0 0}, upload=2341]: Done begin upload segment.
Jul 27 00:24:15.290568944 (2h32m31.071502329s) [noise=true , address=77.90.31.14:4661 , piece_id=354M4K53NIFHG2F5B34YXTAAXVOLXCCCH4SHAE56MPI7HZ33GGDA, seg_pos={0 0}, upload=2341]: Uploading piece...
Jul 27 00:24:15.290880163 (2h32m31.071813545s) [noise=true , address=77.90.31.16:4735 , piece_id=F6TKVMHOYXIRQJ2TIALFJGWBD3KAJP3436JXWQ6ACHYEJ5ARNKJA, seg_pos={0 0}, upload=2341]: Uploading piece...
Jul 27 00:24:15.291096947 (2h32m31.072030331s) [noise=true , address=104.36.82.4:4801 , piece_id=AARTJBSLFRFJNTY64JVS7C4I5WOW3PMUDP3GSCJ4WDDOMZBHJQUQ, seg_pos={0 0}, upload=2341]: Uploading piece...
Jul 27 00:24:15.291376582 (2h32m31.072309958s) [noise=true , address=45.91.116.8:4047 , piece_id=KQAQDBYYPTKLQVHWPFAQPMQAQT3LVG2ZMIZIUT27OVKZID3QA3CQ, seg_pos={0 0}, upload=2341]: Uploading piece...
Jul 27 00:24:15.291546777 (2h32m31.072480163s) [noise=true , address=77.90.31.13:4628 , piece_id=Q6LXQVYH5NLRUHPVR7GEM435RQYJA2XYZYLXRN5CMEOMXXU4B2MQ, seg_pos={0 0}, upload=2341]: Uploading piece...
Jul 27 00:24:15.291737132 (2h32m31.072670515s) [noise=true , address=77.90.31.10:4519 , piece_id=ES7E6WWHHSYR4ZFJ3U56UA2C4ID6E2UO3UTBQMK3MHWXKKF7NTSA, seg_pos={0 0}, upload=2341]: Uploading piece...
Jul 27 00:24:15.292151224 (2h32m31.073084609s) [noise=true , address=45.91.116.12:4226 , piece_id=5VLTPXGXLV3AVI4IRGFEHPSAEFYGCKRZWLUBQA5OK4NQKQQB7FZQ, seg_pos={0 0}, upload=2341]: Uploading piece...
Jul 27 00:24:15.292277278 (2h32m31.073210659s) [noise=true , address=104.36.82.7:4911 , piece_id=FATE6O34GWR4OFDNN3MJWFA2DKTFJF4O7TS3EEVBHC6PYBIPWZDA, seg_pos={0 0}, upload=2341]: Uploading piece...
Jul 27 00:24:15.292593888 (2h32m31.073527263s) [noise=true , address=45.91.116.6:4973 , piece_id=3JKFXNCOWGZDS5DY5RPU2G32DLRY2OAM6KGVTQ6NXO5ENZKYLE2Q, seg_pos={0 0}, upload=2341]: Uploading piece...
Jul 27 00:24:15.292661711 (2h32m31.073595092s) [noise=true , address=45.91.116.13:4154 , piece_id=CMWMJACSKS6XRRCGE3JIVKJ56C7HQMYP7MK3F4OZPQ4XRUL33GUQ, seg_pos={0 0}, upload=2341]: Uploading piece...
Jul 27 00:24:15.292906506 (2h32m31.073839884s) [noise=true , address=45.91.116.5:4297 , piece_id=QHJFDHCZZ7SS5JP3X6TA2DDVNBRL4CX2FT7BFH6S5IWBSKYZ7LNQ, seg_pos={0 0}, upload=2341]: Uploading piece...
Jul 27 00:24:15.293095594 (2h32m31.074028973s) [noise=true , address=45.91.116.9:4261 , piece_id=2Q7BMHN3ZXMRVGWHXTYA7PPQBH4CZ4PKH2EZA6Y3IBJJFP2COH3A, seg_pos={0 0}, upload=2341]: Uploading piece...
Jul 27 00:24:15.293278492 (2h32m31.074211875s) [noise=true , address=45.91.116.4:4937 , piece_id=2VLKTZW4B2BUNWFOGMLRD5NLWAMCA35J2FOMM5GHRPRFTIVOFTIA, seg_pos={0 0}, upload=2341]: Uploading piece...
Jul 27 00:24:15.293527724 (2h32m31.074461109s) [noise=true , address=104.36.82.6:4876 , piece_id=OMCJOJ3BRXAIDOCT4ONBGM7WLMAESXHCM7FRQGUNQSEQ4KNXGFJA, seg_pos={0 0}, upload=2341]: Uploading piece...
Jul 27 00:24:15.293665152 (2h32m31.074598528s) [noise=true , address=77.90.31.10:4518 , piece_id=LGVDHB2LJLE7X3ZQXRAEADD4EOM5XAQJLRX3K3YSQGIMKW7ULFRA, seg_pos={0 0}, upload=2341]: Uploading piece...
Jul 27 00:24:15.293996613 (2h32m31.074929991s) [noise=true , address=45.91.116.7:4009 , piece_id=33K7VJVNW6R3DPD34ASRJQMWN5CFNLBDUTGCBMPKMW7OUZK2VPDQ, seg_pos={0 0}, upload=2341]: Uploading piece...
Jul 27 00:24:15.377667276 (2h32m31.158600652s) [upload=2341]: Next returned nil segment
Jul 27 00:24:15.377684557 (2h32m31.158617932s) [upload=2341]: Waiting for error group managing segments...
Jul 27 00:24:15.381462128 (2h32m31.162395496s) [noise=true , address=45.91.116.6:4973 , piece_id=3JKFXNCOWGZDS5DY5RPU2G32DLRY2OAM6KGVTQ6NXO5ENZKYLE2Q, seg_pos={0 0}, upload=2341]: Done uploading piece. err: <nil>
Jul 27 00:24:15.382306088 (2h32m31.163239457s) [noise=true , address=45.91.116.12:4226 , piece_id=5VLTPXGXLV3AVI4IRGFEHPSAEFYGCKRZWLUBQA5OK4NQKQQB7FZQ, seg_pos={0 0}, upload=2341]: Done uploading piece. err: <nil>
Jul 27 00:24:15.382553700 (2h32m31.163487072s) [noise=true , address=45.91.116.9:4261 , piece_id=2Q7BMHN3ZXMRVGWHXTYA7PPQBH4CZ4PKH2EZA6Y3IBJJFP2COH3A, seg_pos={0 0}, upload=2341]: Done uploading piece. err: <nil>
Jul 27 00:24:15.382934775 (2h32m31.163868154s) [noise=true , address=45.91.116.13:4154 , piece_id=CMWMJACSKS6XRRCGE3JIVKJ56C7HQMYP7MK3F4OZPQ4XRUL33GUQ, seg_pos={0 0}, upload=2341]: Done uploading piece. err: <nil>
Jul 27 00:24:15.383616941 (2h32m31.164550309s) [noise=true , address=45.91.116.5:4297 , piece_id=QHJFDHCZZ7SS5JP3X6TA2DDVNBRL4CX2FT7BFH6S5IWBSKYZ7LNQ, seg_pos={0 0}, upload=2341]: Done uploading piece. err: <nil>
Jul 27 00:24:15.384175789 (2h32m31.165109182s) [noise=true , address=45.91.116.8:4047 , piece_id=KQAQDBYYPTKLQVHWPFAQPMQAQT3LVG2ZMIZIUT27OVKZID3QA3CQ, seg_pos={0 0}, upload=2341]: Done uploading piece. err: <nil>
Jul 27 00:24:15.384717162 (2h32m31.165650542s) [noise=true , address=77.90.31.14:4661 , piece_id=354M4K53NIFHG2F5B34YXTAAXVOLXCCCH4SHAE56MPI7HZ33GGDA, seg_pos={0 0}, upload=2341]: Done uploading piece. err: <nil>
Jul 27 00:24:15.385236419 (2h32m31.16616979s) [noise=true , address=45.91.116.7:4009 , piece_id=33K7VJVNW6R3DPD34ASRJQMWN5CFNLBDUTGCBMPKMW7OUZK2VPDQ, seg_pos={0 0}, upload=2341]: Done uploading piece. err: <nil>
Jul 27 00:24:15.385764587 (2h32m31.166697956s) [noise=true , address=45.91.116.4:4937 , piece_id=2VLKTZW4B2BUNWFOGMLRD5NLWAMCA35J2FOMM5GHRPRFTIVOFTIA, seg_pos={0 0}, upload=2341]: Done uploading piece. err: <nil>
Jul 27 00:24:16.802561849 (2h32m32.583495218s) [noise=true , address=77.90.31.13:4628 , piece_id=Q6LXQVYH5NLRUHPVR7GEM435RQYJA2XYZYLXRN5CMEOMXXU4B2MQ, seg_pos={0 0}, upload=2341]: Done uploading piece. err: <nil>
Jul 27 00:24:45.226162479 (2h33m1.007095865s) [noise=true , address=77.90.31.10:4519 , piece_id=ES7E6WWHHSYR4ZFJ3U56UA2C4ID6E2UO3UTBQMK3MHWXKKF7NTSA, seg_pos={0 0}, upload=2341]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: rpc: dial tcp 77.90.31.10:4519: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:24:45.226264594 (2h33m1.007197978s) [noise=true , address=104.36.82.7:4911 , piece_id=FATE6O34GWR4OFDNN3MJWFA2DKTFJF4O7TS3EEVBHC6PYBIPWZDA, seg_pos={0 0}, upload=2341]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:24:45.226335397 (2h33m1.007268781s) [noise=true , address=104.36.82.6:4876 , piece_id=OMCJOJ3BRXAIDOCT4ONBGM7WLMAESXHCM7FRQGUNQSEQ4KNXGFJA, seg_pos={0 0}, upload=2341]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:54796->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:24:45.226417188 (2h33m1.007350565s) [noise=true , address=77.90.31.10:4518 , piece_id=LGVDHB2LJLE7X3ZQXRAEADD4EOM5XAQJLRX3K3YSQGIMKW7ULFRA, seg_pos={0 0}, upload=2341]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:24:45.226515017 (2h33m1.007448393s) [noise=true , address=77.90.31.16:4735 , piece_id=F6TKVMHOYXIRQJ2TIALFJGWBD3KAJP3436JXWQ6ACHYEJ5ARNKJA, seg_pos={0 0}, upload=2341]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 159.223.143.39:35544->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:24:45.226875599 (2h33m1.007808984s) [noise=true , address=104.36.82.4:4801 , piece_id=AARTJBSLFRFJNTY64JVS7C4I5WOW3PMUDP3GSCJ4WDDOMZBHJQUQ, seg_pos={0 0}, upload=2341]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 159.223.143.39:35016->104.36.82.4:4801: use of closed network connection; context deadline exceeded
Jul 27 00:24:45.226924742 (2h33m1.007858123s) [seg_pos={0 0}, upload=2341]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 27 00:24:45.260403710 (2h33m1.041337086s) [upload=2341]: Done uploading segments...
Jul 27 00:24:45.260436570 (2h33m1.041369954s) [upload=2341]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 30.037870914s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [.........X.X........X...XX...............................X..] duration:
37.270035113s
FAIL (2025-07-27 00:24:35 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 37.270062145s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.033393171s
FAIL (2025-07-27 00:24:26 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
56MCVOLGTGD5SI7H47F7ISMKWQ, seg_pos={0 0}, upload=2331]: Uploading piece...
Jul 27 00:23:26.068393908 (2h31m41.849327276s) [noise=true , address=77.90.31.16:4754 , piece_id=OTMENKECVDUSRPI4E6XOKUR6I5RLFXARL6GMWGLRVADL3UEWO3RA, seg_pos={0 0}, upload=2331]: Uploading piece...
Jul 27 00:23:26.068435197 (2h31m41.849368577s) [noise=true , address=79.127.219.38:4004 , piece_id=A3KLNDYIXZYJSHXDDBM6KDTJA3WBY2PQQVAG3AHUFYUGKMKJWH6Q, seg_pos={0 0}, upload=2331]: Uploading piece...
Jul 27 00:23:26.068934312 (2h31m41.849867697s) [noise=true , address=104.36.82.15:4281 , piece_id=HOLAPLJHZ7ETGQBXXJEH6XJ6DNMIKLO725CVZBMJSCHTNEM2B4SQ, seg_pos={0 0}, upload=2331]: Uploading piece...
Jul 27 00:23:26.069392366 (2h31m41.850325747s) [noise=true , address=79.127.205.234:4618 , piece_id=TNKJNH7KY7VSDZH7F4CX364TY3USEH3X6VOF6ZQNVDZWYEKRJFLQ, seg_pos={0 0}, upload=2331]: Uploading piece...
Jul 27 00:23:26.069716819 (2h31m41.850650187s) [noise=true , address=104.36.82.56:4242 , piece_id=H7SMMJXQDU6XSH4KOJ22CIRNGV6WOMILRSXA4RNTKF46EUMR3B3A, seg_pos={0 0}, upload=2331]: Uploading piece...
Jul 27 00:23:26.069889355 (2h31m41.850822841s) [noise=true , address=77.90.31.15:4727 , piece_id=GEBZJ2R7EBBSUP4KVKAIGPSKTCSODGKMJEVCHBIJSJESPMFYI7MQ, seg_pos={0 0}, upload=2331]: Uploading piece...
Jul 27 00:23:26.070316539 (2h31m41.851249923s) [noise=true , address=104.36.82.44:4888 , piece_id=B62KC75RXRBQ5Q4OS4RLXOSHITLMMTJST52KUV3PRCKBJ33GPFOQ, seg_pos={0 0}, upload=2331]: Uploading piece...
Jul 27 00:23:26.070660719 (2h31m41.851594097s) [noise=true , address=104.36.82.46:4510 , piece_id=TMDPJEIAYODA27N4XYCY6PHLYA5K27HSTA54YD22CPVKT3NONX3A, seg_pos={0 0}, upload=2331]: Uploading piece...
Jul 27 00:23:26.071188191 (2h31m41.852121574s) [noise=true , address=79.127.201.215:4351 , piece_id=Z263YN7U7VBSO6L7NH7K5ALDNUN27BTZDKRROU6PP45E37IGNIUA, seg_pos={0 0}, upload=2331]: Uploading piece...
Jul 27 00:23:26.071581001 (2h31m41.85251438s) [noise=true , address=64.49.13.138:4482 , piece_id=5A3J7FOOV22OBELPOWNH7UJBB5VSDSR45FUBG6ZULZ3P5S7K3AYQ, seg_pos={0 0}, upload=2331]: Uploading piece...
Jul 27 00:23:26.072037758 (2h31m41.852971144s) [noise=true , address=79.127.205.245:4239 , piece_id=AKDBKKB5ISBHKP4BIEOOR4CO6OLQ3SBBRHL43ZULPRQG2LZ23ASA, seg_pos={0 0}, upload=2331]: Uploading piece...
Jul 27 00:23:26.072186947 (2h31m41.853120331s) [noise=true , address=77.90.31.10:4540 , piece_id=ZH2L76FPP24Y36STCI6AJW4JVNABCM4MRW7ENX27JH53XP2RZXIA, seg_pos={0 0}, upload=2331]: Uploading piece...
Jul 27 00:23:26.072608764 (2h31m41.853542147s) [noise=true , address=104.36.82.50:4048 , piece_id=GQUJDQ4EHLBKBA7GTC7KIVQ3A3YE3SKA5POJL543GMA37OETE7ZA, seg_pos={0 0}, upload=2331]: Uploading piece...
Jul 27 00:23:26.073047059 (2h31m41.853980442s) [noise=true , address=79.127.201.214:4321 , piece_id=HWDULUJQG5ZKA5WLVVR2RX6DOMR7SLLUZGQKHO7UTLFL2MTONQRA, seg_pos={0 0}, upload=2331]: Uploading piece...
Jul 27 00:23:26.073644658 (2h31m41.854578032s) [noise=true , address=104.36.82.47:4517 , piece_id=X3RVSICOB7DRHEMMRUG6Q7GG3XR5DKZYHMBAXRHHCDY3NWJUTWRA, seg_pos={0 0}, upload=2331]: Uploading piece...
Jul 27 00:23:26.074244739 (2h31m41.855178123s) [noise=true , address=104.36.82.54:4171 , piece_id=HA4FPTMKKADZFA4LHBKJWIWVEGRINAQ253M2JIO3K4QGNNJTGEWA, seg_pos={0 0}, upload=2331]: Uploading piece...
Jul 27 00:23:26.147711829 (2h31m41.928645211s) [upload=2331]: Next returned nil segment
Jul 27 00:23:26.148916453 (2h31m41.929849843s) [upload=2331]: Waiting for error group managing segments...
Jul 27 00:23:26.197757268 (2h31m41.978690638s) [noise=true , address=77.90.31.15:4727 , piece_id=GEBZJ2R7EBBSUP4KVKAIGPSKTCSODGKMJEVCHBIJSJESPMFYI7MQ, seg_pos={0 0}, upload=2331]: Done uploading piece. err: <nil>
Jul 27 00:23:26.275108732 (2h31m42.05604211s) [noise=true , address=79.127.219.38:4004 , piece_id=A3KLNDYIXZYJSHXDDBM6KDTJA3WBY2PQQVAG3AHUFYUGKMKJWH6Q, seg_pos={0 0}, upload=2331]: Done uploading piece. err: <nil>
Jul 27 00:23:26.653390226 (2h31m42.434323603s) [noise=true , address=79.127.205.246:4266 , piece_id=DE7LHDVHS34DSHZA6RYUQ3PVP656MCVOLGTGD5SI7H47F7ISMKWQ, seg_pos={0 0}, upload=2331]: Done uploading piece. err: <nil>
Jul 27 00:23:26.664941515 (2h31m42.445874883s) [noise=true , address=79.127.205.245:4239 , piece_id=AKDBKKB5ISBHKP4BIEOOR4CO6OLQ3SBBRHL43ZULPRQG2LZ23ASA, seg_pos={0 0}, upload=2331]: Done uploading piece. err: <nil>
Jul 27 00:23:26.701941806 (2h31m42.482875179s) [noise=true , address=79.127.205.234:4618 , piece_id=TNKJNH7KY7VSDZH7F4CX364TY3USEH3X6VOF6ZQNVDZWYEKRJFLQ, seg_pos={0 0}, upload=2331]: Done uploading piece. err: <nil>
Jul 27 00:23:30.267346655 (2h31m46.048280039s) [noise=true , address=79.127.201.214:4321 , piece_id=HWDULUJQG5ZKA5WLVVR2RX6DOMR7SLLUZGQKHO7UTLFL2MTONQRA, seg_pos={0 0}, upload=2331]: Done uploading piece. err: <nil>
Jul 27 00:23:32.394608110 (2h31m48.1755415s) [noise=true , address=64.49.13.138:4482 , piece_id=5A3J7FOOV22OBELPOWNH7UJBB5VSDSR45FUBG6ZULZ3P5S7K3AYQ, seg_pos={0 0}, upload=2331]: Done uploading piece. err: <nil>
Jul 27 00:23:38.279720172 (2h31m54.060653559s) [noise=true , address=45.91.116.14:4206 , piece_id=VS7D3DL6CV6WZ5UKEBW4ZMIPRNHTDVYTEMVL6BL7HSQ2KAECG5EA, seg_pos={0 0}, upload=2331]: Done uploading piece. err: <nil>
Jul 27 00:24:01.540097748 (2h32m17.321031132s) [noise=true , address=79.127.201.215:4351 , piece_id=Z263YN7U7VBSO6L7NH7K5ALDNUN27BTZDKRROU6PP45E37IGNIUA, seg_pos={0 0}, upload=2331]: Done uploading piece. err: <nil>
Jul 27 00:24:25.973451469 (2h32m41.754384845s) [noise=true , address=104.36.82.15:4281 , piece_id=HOLAPLJHZ7ETGQBXXJEH6XJ6DNMIKLO725CVZBMJSCHTNEM2B4SQ, seg_pos={0 0}, upload=2331]: Done uploading piece. err: ecclient: upload failed (node:12hDXeAm4tomp1gAkBNjM8x77V76yiruYQmPQ3BZEYPLpJzigo2, address:104.36.82.15:4281): protocol: rpc: dial tcp 104.36.82.15:4281: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:24:25.973519077 (2h32m41.754452454s) [noise=true , address=77.90.31.10:4540 , piece_id=ZH2L76FPP24Y36STCI6AJW4JVNABCM4MRW7ENX27JH53XP2RZXIA, seg_pos={0 0}, upload=2331]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:24:25.973593522 (2h32m41.754526899s) [noise=true , address=104.36.82.50:4048 , piece_id=GQUJDQ4EHLBKBA7GTC7KIVQ3A3YE3SKA5POJL543GMA37OETE7ZA, seg_pos={0 0}, upload=2331]: Done uploading piece. err: ecclient: upload failed (node:1rJd5FahWjKfvJstcoLE5uJGgtVj2uZirPpCJxbfGPGQ3j4qxC, address:104.36.82.50:4048): protocol: write tcp 159.223.143.39:60142->104.36.82.50:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:24:25.973656555 (2h32m41.754589931s) [noise=true , address=104.36.82.56:4242 , piece_id=H7SMMJXQDU6XSH4KOJ22CIRNGV6WOMILRSXA4RNTKF46EUMR3B3A, seg_pos={0 0}, upload=2331]: Done uploading piece. err: ecclient: upload failed (node:1Q615w5XWJT1ownaJL1JFqyqntzMj6fgNieAfR6z421YCHuR5r, address:104.36.82.56:4242): protocol: rpc: dial tcp 104.36.82.56:4242: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:24:25.973849672 (2h32m41.754783048s) [noise=true , address=104.36.82.44:4888 , piece_id=B62KC75RXRBQ5Q4OS4RLXOSHITLMMTJST52KUV3PRCKBJ33GPFOQ, seg_pos={0 0}, upload=2331]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: write tcp 159.223.143.39:50010->104.36.82.44:4888: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:24:25.973909573 (2h32m41.754842944s) [noise=true , address=104.36.82.54:4171 , piece_id=HA4FPTMKKADZFA4LHBKJWIWVEGRINAQ253M2JIO3K4QGNNJTGEWA, seg_pos={0 0}, upload=2331]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: write tcp 159.223.143.39:55640->104.36.82.54:4171: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:24:25.973940294 (2h32m41.754873673s) [noise=true , address=77.90.31.16:4754 , piece_id=OTMENKECVDUSRPI4E6XOKUR6I5RLFXARL6GMWGLRVADL3UEWO3RA, seg_pos={0 0}, upload=2331]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:24:25.974020126 (2h32m41.754953512s) [noise=true , address=104.36.82.46:4510 , piece_id=TMDPJEIAYODA27N4XYCY6PHLYA5K27HSTA54YD22CPVKT3NONX3A, seg_pos={0 0}, upload=2331]: Done uploading piece. err: ecclient: upload failed (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ, address:104.36.82.46:4510): protocol: write tcp 159.223.143.39:42322->104.36.82.46:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:24:25.974069289 (2h32m41.755002661s) [noise=true , address=104.36.82.47:4517 , piece_id=X3RVSICOB7DRHEMMRUG6Q7GG3XR5DKZYHMBAXRHHCDY3NWJUTWRA, seg_pos={0 0}, upload=2331]: Done uploading piece. err: ecclient: upload failed (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, address:104.36.82.47:4517): protocol: write tcp 159.223.143.39:49678->104.36.82.47:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:24:25.974111876 (2h32m41.755045248s) [seg_pos={0 0}, upload=2331]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Jul 27 00:24:26.001491819 (2h32m41.782425201s) [upload=2331]: Done uploading segments...
Jul 27 00:24:26.001524618 (2h32m41.782457986s) [upload=2331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 1m0.033431376s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [X...............X..X......X.XX....X...X.................XX..] duration:
1.839939307s
FAIL (2025-07-27 00:24:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.839971886s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
39 out of 60 is failed (threshold=3) [XXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.] duration:
2.497043876s
FAIL (2025-07-27 00:23:58 UTC)
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 2.497143537s
core
uplink
us1
vivint
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
40 out of 60 is failed (threshold=3) [XXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX] duration:
30.028014755s
FAIL (2025-07-27 00:23:40 UTC)
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 30.028043855s
core
uplink
us1
vivint
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
43 out of 60 is failed (threshold=3) [XXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXX] duration:
30.032367125s
FAIL (2025-07-27 00:23:35 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
load enough pieces (needed at least 13 but got 10)
Jul 27 00:23:05.830875957 (2h31m21.611809333s) [upload=2325]: Starting upload
Jul 27 00:23:05.830976568 (2h31m21.611909942s) [upload=2325]: Done starting upload
Jul 27 00:23:05.831142029 (2h31m21.612075403s) [upload=2325]: Uploading segments...
Jul 27 00:23:05.831520163 (2h31m21.61245353s) [upload=2325]: Got next segment. Inline: false
Jul 27 00:23:05.831589712 (2h31m21.612523088s) [upload=2325]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 27 00:23:05.831595331 (2h31m21.612528701s) [upload=2325]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 27 00:23:05.859493989 (2h31m21.640427375s) [seg_pos={0 0}, upload=2325]: Begin upload segment...
Jul 27 00:23:05.859548834 (2h31m21.640482209s) [seg_pos={0 0}, upload=2325]: Done begin upload segment.
Jul 27 00:23:05.859597857 (2h31m21.640531241s) [noise=true , address=45.91.116.7:4009 , piece_id=PC4BDAPBLCJO7UJEQMQUV23TYFPUEF3GCO5KGBUWBAF5OULRYWWA, seg_pos={0 0}, upload=2325]: Uploading piece...
Jul 27 00:23:05.859628700 (2h31m21.640562082s) [noise=true , address=104.36.82.6:4875 , piece_id=R3L46YYQCH74UYBV6FJUENTZXSBEO75W7NUZVB2IEUEXSCHNSZWQ, seg_pos={0 0}, upload=2325]: Uploading piece...
Jul 27 00:23:05.859697244 (2h31m21.640630612s) [noise=true , address=45.91.116.10:4083 , piece_id=JHWXXNPRVODBPGIYGVSWTHFKK43FTUJKUMUYFEDJO4LVCUW3ATDQ, seg_pos={0 0}, upload=2325]: Uploading piece...
Jul 27 00:23:05.860273748 (2h31m21.641207133s) [noise=true , address=77.90.31.13:4627 , piece_id=GBFZPOPW44JHI7GXWBVMT7T7U7GJ7ZQY5BMLBC4U5NFT36VJ3S6A, seg_pos={0 0}, upload=2325]: Uploading piece...
Jul 27 00:23:05.860328404 (2h31m21.641261772s) [noise=true , address=45.91.116.14:4189 , piece_id=HECENZE2ESPYKNG32TMADTVMBBKPBAYASHSSWC4EUJK3K3U3MP6A, seg_pos={0 0}, upload=2325]: Uploading piece...
Jul 27 00:23:05.860841565 (2h31m21.64177496s) [noise=true , address=104.36.82.6:4874 , piece_id=GSERE7IZD3WUOAL4BBW2GWKNMMQWRM3HHYBDXRALEEJLAIQEZ6XA, seg_pos={0 0}, upload=2325]: Uploading piece...
Jul 27 00:23:05.860853636 (2h31m21.641787023s) [noise=true , address=77.90.31.18:4805 , piece_id=4FLYNQHNAEIVPRPXFUJPSOFEJUMRHH6LSCOIVR7AZSZ4W7KAZZEQ, seg_pos={0 0}, upload=2325]: Uploading piece...
Jul 27 00:23:05.861242275 (2h31m21.642175655s) [noise=true , address=77.90.31.17:4771 , piece_id=BLV6ZGDEBKCWTELYBFORKGF2A2IMW5ILBB6CF2S2QHBOTALKW4BQ, seg_pos={0 0}, upload=2325]: Uploading piece...
Jul 27 00:23:05.861317433 (2h31m21.642250803s) [noise=true , address=104.36.82.5:4838 , piece_id=5RLMP5ZARSNRBOKXU5RYA65FW6BRKKWO7JVZGKKOH3NYWOOCFQRA, seg_pos={0 0}, upload=2325]: Uploading piece...
Jul 27 00:23:05.861624060 (2h31m21.642557447s) [noise=true , address=77.90.31.10:4518 , piece_id=JNOXAEXTO2XR5R262KIHPQKXTX4J5LWBYVCHMY4RJINLOOA775TQ, seg_pos={0 0}, upload=2325]: Uploading piece...
Jul 27 00:23:05.861970118 (2h31m21.642903501s) [noise=true , address=104.36.82.10:4077 , piece_id=L6TYU6DTHHHBM3AGOXE5IY5JFI2CQ53UYIR7MZMBGWU3N5G2AVVQ, seg_pos={0 0}, upload=2325]: Uploading piece...
Jul 27 00:23:05.862371652 (2h31m21.643305032s) [noise=true , address=77.90.31.19:4841 , piece_id=Q2TOOY7WLFPLAUS4SF3Y2VARO6FKDLM5KOG4N3GOZIPQVCO43DPQ, seg_pos={0 0}, upload=2325]: Uploading piece...
Jul 27 00:23:05.863110035 (2h31m21.644043404s) [noise=true , address=104.36.82.12:4149 , piece_id=SSDAXUUCREBIUHE7SJ3UPMLPL45VUBFUVPUIBFYXTQEK553F4AAA, seg_pos={0 0}, upload=2325]: Uploading piece...
Jul 27 00:23:05.863579173 (2h31m21.644512549s) [noise=true , address=45.91.116.9:4262 , piece_id=DHLZBDPP5N23JLGLGB4S4M57XU2WHEDUW4NIK33J2YXSTQBFCO5Q, seg_pos={0 0}, upload=2325]: Uploading piece...
Jul 27 00:23:05.863673501 (2h31m21.644606871s) [noise=true , address=104.36.82.7:4911 , piece_id=ABESGXHYNDJ7OL27XBRDVNB3MCPOF3CFVZV6X2A7KRBTNGWCOY3Q, seg_pos={0 0}, upload=2325]: Uploading piece...
Jul 27 00:23:05.864361977 (2h31m21.645295363s) [noise=true , address=77.90.31.14:4663 , piece_id=6DY3BNYAGEHJTTREC7OLQ2AF37MKL7IVYUV7ERK4ZRI73MVTWU7Q, seg_pos={0 0}, upload=2325]: Uploading piece...
Jul 27 00:23:05.944508944 (2h31m21.725442325s) [upload=2325]: Next returned nil segment
Jul 27 00:23:05.944531926 (2h31m21.72546531s) [upload=2325]: Waiting for error group managing segments...
Jul 27 00:23:05.949015550 (2h31m21.729948926s) [noise=true , address=45.91.116.7:4009 , piece_id=PC4BDAPBLCJO7UJEQMQUV23TYFPUEF3GCO5KGBUWBAF5OULRYWWA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil>
Jul 27 00:23:05.952473241 (2h31m21.733406612s) [noise=true , address=45.91.116.10:4083 , piece_id=JHWXXNPRVODBPGIYGVSWTHFKK43FTUJKUMUYFEDJO4LVCUW3ATDQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil>
Jul 27 00:23:05.952908571 (2h31m21.73384194s) [noise=true , address=77.90.31.14:4663 , piece_id=6DY3BNYAGEHJTTREC7OLQ2AF37MKL7IVYUV7ERK4ZRI73MVTWU7Q, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil>
Jul 27 00:23:05.953379182 (2h31m21.734312558s) [noise=true , address=45.91.116.14:4189 , piece_id=HECENZE2ESPYKNG32TMADTVMBBKPBAYASHSSWC4EUJK3K3U3MP6A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil>
Jul 27 00:23:05.953872699 (2h31m21.734806069s) [noise=true , address=45.91.116.9:4262 , piece_id=DHLZBDPP5N23JLGLGB4S4M57XU2WHEDUW4NIK33J2YXSTQBFCO5Q, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil>
Jul 27 00:23:05.954263594 (2h31m21.735196968s) [noise=true , address=77.90.31.18:4805 , piece_id=4FLYNQHNAEIVPRPXFUJPSOFEJUMRHH6LSCOIVR7AZSZ4W7KAZZEQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil>
Jul 27 00:23:07.351282983 (2h31m23.132216363s) [noise=true , address=77.90.31.19:4841 , piece_id=Q2TOOY7WLFPLAUS4SF3Y2VARO6FKDLM5KOG4N3GOZIPQVCO43DPQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil>
Jul 27 00:23:07.370174561 (2h31m23.151107923s) [noise=true , address=77.90.31.13:4627 , piece_id=GBFZPOPW44JHI7GXWBVMT7T7U7GJ7ZQY5BMLBC4U5NFT36VJ3S6A, seg_pos={0 0}, upload=2325]: Done uploading piece. err: <nil>
Jul 27 00:23:35.801690340 (2h31m51.582623727s) [noise=true , address=104.36.82.10:4077 , piece_id=L6TYU6DTHHHBM3AGOXE5IY5JFI2CQ53UYIR7MZMBGWU3N5G2AVVQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:23:35.801789541 (2h31m51.582722919s) [noise=true , address=77.90.31.17:4771 , piece_id=BLV6ZGDEBKCWTELYBFORKGF2A2IMW5ILBB6CF2S2QHBOTALKW4BQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:23:35.801851272 (2h31m51.58278465s) [noise=true , address=104.36.82.7:4911 , piece_id=ABESGXHYNDJ7OL27XBRDVNB3MCPOF3CFVZV6X2A7KRBTNGWCOY3Q, seg_pos={0 0}, upload=2325]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:41292->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:23:35.801921382 (2h31m51.582854761s) [noise=true , address=104.36.82.12:4149 , piece_id=SSDAXUUCREBIUHE7SJ3UPMLPL45VUBFUVPUIBFYXTQEK553F4AAA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:23:35.801960525 (2h31m51.582893887s) [noise=true , address=104.36.82.5:4838 , piece_id=5RLMP5ZARSNRBOKXU5RYA65FW6BRKKWO7JVZGKKOH3NYWOOCFQRA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 159.223.143.39:59918->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:23:35.801983053 (2h31m51.58291643s) [noise=true , address=104.36.82.6:4874 , piece_id=GSERE7IZD3WUOAL4BBW2GWKNMMQWRM3HHYBDXRALEEJLAIQEZ6XA, seg_pos={0 0}, upload=2325]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:23:35.802023711 (2h31m51.582957089s) [noise=true , address=77.90.31.10:4518 , piece_id=JNOXAEXTO2XR5R262KIHPQKXTX4J5LWBYVCHMY4RJINLOOA775TQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:23:35.802048687 (2h31m51.582982065s) [noise=true , address=104.36.82.6:4875 , piece_id=R3L46YYQCH74UYBV6FJUENTZXSBEO75W7NUZVB2IEUEXSCHNSZWQ, seg_pos={0 0}, upload=2325]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 159.223.143.39:53440->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:23:35.802082296 (2h31m51.583015681s) [seg_pos={0 0}, upload=2325]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Jul 27 00:23:35.830966119 (2h31m51.611899487s) [upload=2325]: Done uploading segments...
Jul 27 00:23:35.830993756 (2h31m51.611927137s) [upload=2325]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 30.032387822s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [XX...............X..X......X.XX....X...X.................XX.] duration:
9.561507357s
FAIL (2025-07-27 00:23:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 9.561556497s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.031647854s
FAIL (2025-07-27 00:23:25 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
item: *pb.BatchRequestItem_ObjectBegin
Jul 27 00:22:25.898161176 (2h30m41.679094553s) [upload=2313]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 27 00:22:25.940641206 (2h30m41.721574593s) [seg_pos={0 0}, upload=2313]: Begin upload segment...
Jul 27 00:22:25.940716081 (2h30m41.721649443s) [seg_pos={0 0}, upload=2313]: Done begin upload segment.
Jul 27 00:22:25.940767132 (2h30m41.721700518s) [noise=true , address=104.36.82.19:4431 , piece_id=3HZC5ON4OKUMVROACB4CSFHB6B64Z6CH7YSYSNKZ7JLH3V7X4JWA, seg_pos={0 0}, upload=2313]: Uploading piece...
Jul 27 00:22:25.941254472 (2h30m41.722187859s) [noise=true , address=104.36.82.52:4096 , piece_id=RB3SRXV6TURCZC24MNNKYNRNHAK7HTZ5IVQKAU5T432CGETFFOLQ, seg_pos={0 0}, upload=2313]: Uploading piece...
Jul 27 00:22:25.941669697 (2h30m41.722603077s) [noise=true , address=77.90.31.21:4907 , piece_id=VGN4MO5V6AGFAFHXLPMSGAC23KET27OR3SMX54NVVC523ZBAKUNQ, seg_pos={0 0}, upload=2313]: Uploading piece...
Jul 27 00:22:25.942027107 (2h30m41.722960497s) [noise=true , address=64.49.13.137:4502 , piece_id=REKSKHRQUS6VGKYDEJKC73NRZCY6GVNOGMY23WW5KO6SWBTWZL4Q, seg_pos={0 0}, upload=2313]: Uploading piece...
Jul 27 00:22:25.942390382 (2h30m41.723323763s) [noise=true , address=79.127.220.101:4048 , piece_id=CFAHURCN6KJ4JW4HXQNI74BQ3GIJUOJWZDVBKEV5HD4H6OAIMX5A, seg_pos={0 0}, upload=2313]: Uploading piece...
Jul 27 00:22:25.942698295 (2h30m41.723631674s) [noise=true , address=64.49.13.140:4508 , piece_id=G6QOJIAJEIOUZGE23GPOW23ESGM5AZXBEGVYC2BKD3JAII52RZJA, seg_pos={0 0}, upload=2313]: Uploading piece...
Jul 27 00:22:25.942981866 (2h30m41.723915246s) [noise=true , address=64.49.13.26:4953 , piece_id=UVEUWX2HWLBAPGKMPVTZXDWE46655QSOSWJYNBATEAR2DNCNTHCQ, seg_pos={0 0}, upload=2313]: Uploading piece...
Jul 27 00:22:25.943384935 (2h30m41.724318317s) [noise=true , address=64.49.13.139:4478 , piece_id=QCUOTMGT547UX5OFD5J3DGPV4II4VMTX46UEVIJQXLM4XVDHB7NA, seg_pos={0 0}, upload=2313]: Uploading piece...
Jul 27 00:22:25.943582481 (2h30m41.724515884s) [noise=true , address=45.91.116.9:4277 , piece_id=VFV26NRL4V63FM6OVVO7YSV2HEKVX3IMNLYVADFGQ53PVX5X5B2Q, seg_pos={0 0}, upload=2313]: Uploading piece...
Jul 27 00:22:25.943729340 (2h30m41.724662729s) [noise=true , address=79.127.219.38:4018 , piece_id=DWI35RTF2QE365DMZUU7WFEJRUHIRZEGQEOHZBBQNLIJ3ZFKQSSQ, seg_pos={0 0}, upload=2313]: Uploading piece...
Jul 27 00:22:25.944218928 (2h30m41.725152351s) [noise=true , address=79.127.205.235:4691 , piece_id=NHXU5I6RHDFMMQD5FIF2VBPA6Q2KID6LOIYJPTJTQALAOLJ7DDTA, seg_pos={0 0}, upload=2313]: Uploading piece...
Jul 27 00:22:25.944620429 (2h30m41.725553816s) [noise=true , address=79.127.205.233:4646 , piece_id=FB3DEPJHM3A7USMLM22MPDMJYZIY4BSQXXEZNPNWJF4LLCCCL7MQ, seg_pos={0 0}, upload=2313]: Uploading piece...
Jul 27 00:22:25.945039019 (2h30m41.725972416s) [noise=true , address=104.36.82.46:4501 , piece_id=62HAXOPLI344F32UV2ZRGEOQQW5LAKF6W3RKRKEKU6YBGY6R2POA, seg_pos={0 0}, upload=2313]: Uploading piece...
Jul 27 00:22:25.945435137 (2h30m41.726368527s) [noise=true , address=104.36.82.56:4258 , piece_id=ABG5XOANCNDLJ6TICN7NRRZTXXGYBINDKXKN64KXNCY3BIV56Z3Q, seg_pos={0 0}, upload=2313]: Uploading piece...
Jul 27 00:22:25.945798530 (2h30m41.72673192s) [noise=true , address=64.49.13.138:4483 , piece_id=X52L3JLMDKSHGDMV2KKZBFKAC7NYYHZSEHPGMFXKKAUPHX3SLVPQ, seg_pos={0 0}, upload=2313]: Uploading piece...
Jul 27 00:22:25.946174758 (2h30m41.727108146s) [noise=true , address=45.91.116.12:4231 , piece_id=QV7RSQIL3KQWGBLYYOEVZU55FBIYPYPQTQ2XHOGLVMPWRPTYGZ5A, seg_pos={0 0}, upload=2313]: Uploading piece...
Jul 27 00:22:25.946547191 (2h30m41.727480579s) [noise=true , address=77.90.31.15:4722 , piece_id=GLXCKILWSMI37DTREARWYCST7PUBYJEDT3CCJGNMYDY6XKR3JTRQ, seg_pos={0 0}, upload=2313]: Uploading piece...
Jul 27 00:22:25.951701239 (2h30m41.732634631s) [noise=true , address=79.127.205.245:4224 , piece_id=YCYZL46TMOVBGYSY2HRPGHERNWBDLJWJMRSWUTJLOIQN6ODM7YHA, seg_pos={0 0}, upload=2313]: Uploading piece...
Jul 27 00:22:26.067329858 (2h30m41.848263222s) [upload=2313]: Next returned nil segment
Jul 27 00:22:26.067353897 (2h30m41.848287264s) [upload=2313]: Waiting for error group managing segments...
Jul 27 00:22:26.069021324 (2h30m41.849954711s) [noise=true , address=79.127.220.101:4048 , piece_id=CFAHURCN6KJ4JW4HXQNI74BQ3GIJUOJWZDVBKEV5HD4H6OAIMX5A, seg_pos={0 0}, upload=2313]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000
Jul 27 00:22:26.074424607 (2h30m41.855357969s) [noise=true , address=79.127.219.38:4018 , piece_id=DWI35RTF2QE365DMZUU7WFEJRUHIRZEGQEOHZBBQNLIJ3ZFKQSSQ, seg_pos={0 0}, upload=2313]: Done uploading piece. err: <nil>
Jul 27 00:22:26.109637102 (2h30m41.89057048s) [noise=true , address=77.90.31.15:4722 , piece_id=GLXCKILWSMI37DTREARWYCST7PUBYJEDT3CCJGNMYDY6XKR3JTRQ, seg_pos={0 0}, upload=2313]: Done uploading piece. err: <nil>
Jul 27 00:22:26.568493158 (2h30m42.349426536s) [noise=true , address=79.127.205.235:4691 , piece_id=NHXU5I6RHDFMMQD5FIF2VBPA6Q2KID6LOIYJPTJTQALAOLJ7DDTA, seg_pos={0 0}, upload=2313]: Done uploading piece. err: <nil>
Jul 27 00:22:26.617213670 (2h30m42.398147053s) [noise=true , address=79.127.205.233:4646 , piece_id=FB3DEPJHM3A7USMLM22MPDMJYZIY4BSQXXEZNPNWJF4LLCCCL7MQ, seg_pos={0 0}, upload=2313]: Done uploading piece. err: <nil>
Jul 27 00:22:26.629366290 (2h30m42.410299667s) [noise=true , address=79.127.205.245:4224 , piece_id=YCYZL46TMOVBGYSY2HRPGHERNWBDLJWJMRSWUTJLOIQN6ODM7YHA, seg_pos={0 0}, upload=2313]: Done uploading piece. err: <nil>
Jul 27 00:22:26.642448415 (2h30m42.423381805s) [noise=true , address=64.49.13.137:4502 , piece_id=REKSKHRQUS6VGKYDEJKC73NRZCY6GVNOGMY23WW5KO6SWBTWZL4Q, seg_pos={0 0}, upload=2313]: Done uploading piece. err: <nil>
Jul 27 00:22:27.151769117 (2h30m42.9327025s) [noise=true , address=45.91.116.9:4277 , piece_id=VFV26NRL4V63FM6OVVO7YSV2HEKVX3IMNLYVADFGQ53PVX5X5B2Q, seg_pos={0 0}, upload=2313]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000
Jul 27 00:22:28.290134394 (2h30m44.071067776s) [noise=true , address=64.49.13.140:4508 , piece_id=G6QOJIAJEIOUZGE23GPOW23ESGM5AZXBEGVYC2BKD3JAII52RZJA, seg_pos={0 0}, upload=2313]: Done uploading piece. err: <nil>
Jul 27 00:22:29.540568553 (2h30m45.321501939s) [noise=true , address=45.91.116.12:4231 , piece_id=QV7RSQIL3KQWGBLYYOEVZU55FBIYPYPQTQ2XHOGLVMPWRPTYGZ5A, seg_pos={0 0}, upload=2313]: Done uploading piece. err: <nil>
Jul 27 00:22:31.674930017 (2h30m47.455863397s) [noise=true , address=64.49.13.26:4953 , piece_id=UVEUWX2HWLBAPGKMPVTZXDWE46655QSOSWJYNBATEAR2DNCNTHCQ, seg_pos={0 0}, upload=2313]: Done uploading piece. err: <nil>
Jul 27 00:22:31.791830817 (2h30m47.572764186s) [noise=true , address=64.49.13.138:4483 , piece_id=X52L3JLMDKSHGDMV2KKZBFKAC7NYYHZSEHPGMFXKKAUPHX3SLVPQ, seg_pos={0 0}, upload=2313]: Done uploading piece. err: <nil>
Jul 27 00:22:33.069396553 (2h30m48.850329917s) [noise=true , address=64.49.13.139:4478 , piece_id=QCUOTMGT547UX5OFD5J3DGPV4II4VMTX46UEVIJQXLM4XVDHB7NA, seg_pos={0 0}, upload=2313]: Done uploading piece. err: <nil>
Jul 27 00:23:25.867516117 (2h31m41.648449487s) [noise=true , address=104.36.82.56:4258 , piece_id=ABG5XOANCNDLJ6TICN7NRRZTXXGYBINDKXKN64KXNCY3BIV56Z3Q, seg_pos={0 0}, upload=2313]: Done uploading piece. err: ecclient: upload failed (node:12g7DdMjQHyYTD9hRvixfNedxyPr6q6uE2rLdEbhQ1MGANRMDm3, address:104.36.82.56:4258): protocol: rpc: dial tcp 104.36.82.56:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:23:25.867575536 (2h31m41.648508902s) [noise=true , address=104.36.82.46:4501 , piece_id=62HAXOPLI344F32UV2ZRGEOQQW5LAKF6W3RKRKEKU6YBGY6R2POA, seg_pos={0 0}, upload=2313]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:104.36.82.46:4501): protocol: write tcp 159.223.143.39:47734->104.36.82.46:4501: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:23:25.867621495 (2h31m41.648554865s) [noise=true , address=77.90.31.21:4907 , piece_id=VGN4MO5V6AGFAFHXLPMSGAC23KET27OR3SMX54NVVC523ZBAKUNQ, seg_pos={0 0}, upload=2313]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: rpc: dial tcp 77.90.31.21:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:23:25.867711307 (2h31m41.648644687s) [noise=true , address=104.36.82.52:4096 , piece_id=RB3SRXV6TURCZC24MNNKYNRNHAK7HTZ5IVQKAU5T432CGETFFOLQ, seg_pos={0 0}, upload=2313]: Done uploading piece. err: ecclient: upload failed (node:1izNXqYvnRLt96tcnR1TXt29qcxZWEgACmHuJ6CikSgLbAzxLL, address:104.36.82.52:4096): protocol: rpc: dial tcp 104.36.82.52:4096: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:23:25.867803587 (2h31m41.64873695s) [noise=true , address=104.36.82.19:4431 , piece_id=3HZC5ON4OKUMVROACB4CSFHB6B64Z6CH7YSYSNKZ7JLH3V7X4JWA, seg_pos={0 0}, upload=2313]: Done uploading piece. err: ecclient: upload failed (node:12E2efGqgEca5TeZV1bWeH9fcQqrmdNwxuU6kMH26McLkLdxNGR, address:104.36.82.19:4431): protocol: write tcp 159.223.143.39:39806->104.36.82.19:4431: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:23:25.867837251 (2h31m41.648770639s) [seg_pos={0 0}, upload=2313]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 27 00:23:25.896823463 (2h31m41.677756839s) [upload=2313]: Done uploading segments...
Jul 27 00:23:25.896860064 (2h31m41.677793443s) [upload=2313]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 1m0.031668044s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [..........X.X........X...XX...............................X.] duration:
28.582429707s
FAIL (2025-07-27 00:23:25 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 28.582454163s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [XXX...............X..X......X.XX....X...X.................XX] duration:
30.001531855s
FAIL (2025-07-27 00:23:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.00154849s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
43 out of 60 is failed (threshold=3) [XXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXX] duration:
30.026156574s
FAIL (2025-07-27 00:23:05 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
WE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:58496->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:22:06.992951239 (2h30m22.773884626s) [seg_pos={0 0}, upload=2302]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 27 00:22:07.017432150 (2h30m22.798365536s) [upload=2302]: Done uploading segments...
Jul 27 00:22:07.017466365 (2h30m22.79839975s) [upload=2302]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Jul 27 00:22:35.735772116 (2h30m51.516705485s) [upload=2316]: Starting upload
Jul 27 00:22:35.735889547 (2h30m51.516822913s) [upload=2316]: Done starting upload
Jul 27 00:22:35.736136946 (2h30m51.517070315s) [upload=2316]: Uploading segments...
Jul 27 00:22:35.736579374 (2h30m51.517512741s) [upload=2316]: Got next segment. Inline: false
Jul 27 00:22:35.736649272 (2h30m51.517582638s) [upload=2316]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 27 00:22:35.736656057 (2h30m51.517589425s) [upload=2316]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 27 00:22:35.773528464 (2h30m51.554461842s) [seg_pos={0 0}, upload=2316]: Begin upload segment...
Jul 27 00:22:35.773571359 (2h30m51.554504724s) [seg_pos={0 0}, upload=2316]: Done begin upload segment.
Jul 27 00:22:35.773607061 (2h30m51.554540439s) [noise=true , address=104.36.82.2:4731 , piece_id=TAHBPPGVULUOM3KNMPUWBGLHUJMETCHHBPWZSP5VRFG3GSTEZSGA, seg_pos={0 0}, upload=2316]: Uploading piece...
Jul 27 00:22:35.774052025 (2h30m51.554985399s) [noise=true , address=77.90.31.21:4878 , piece_id=27RVSTANPTACCVI7UO2LIM2K46SIGQX4A47X7AHKZP2KQPK5XHUA, seg_pos={0 0}, upload=2316]: Uploading piece...
Jul 27 00:22:35.774691559 (2h30m51.555624941s) [noise=true , address=45.91.116.11:4119 , piece_id=V4BNG3X6KOITLDNCFWT6XBDZOJCYDXO5H3N6AJ7MKKF4NXMP72AQ, seg_pos={0 0}, upload=2316]: Uploading piece...
Jul 27 00:22:35.776464301 (2h30m51.557397688s) [noise=true , address=77.90.31.10:4517 , piece_id=4TEO72K4T5IBB3KAYDJ7XA3BMPSOIX4FGLOSM3AYBDVSKRVPGX7A, seg_pos={0 0}, upload=2316]: Uploading piece...
Jul 27 00:22:35.776835877 (2h30m51.55776926s) [noise=true , address=104.36.82.16:4295 , piece_id=3CNUVGW4XWZB4PMYKNWVKHKNRQH5M4XYWLULZ6HPY4GYBJBUYWIQ, seg_pos={0 0}, upload=2316]: Uploading piece...
Jul 27 00:22:35.777053841 (2h30m51.557987254s) [noise=true , address=77.90.31.14:4663 , piece_id=S36NSRGS7IGRKBIOKHLZGMZLIKMXZRSCVZS5WJZIIP7FLGONKLHA, seg_pos={0 0}, upload=2316]: Uploading piece...
Jul 27 00:22:35.777214768 (2h30m51.558148154s) [noise=true , address=104.36.82.14:4223 , piece_id=364455LQ6FYYIA6II4JS7PKFZM7422U2B2KD4DD3VGYVJPQ2YZWA, seg_pos={0 0}, upload=2316]: Uploading piece...
Jul 27 00:22:35.777525863 (2h30m51.558459249s) [noise=true , address=45.91.116.10:4082 , piece_id=OA25WWKBZAQPR7NOM3PFIOR3L75WUAJ2WDSDFLTUOHWPEFQIXL3A, seg_pos={0 0}, upload=2316]: Uploading piece...
Jul 27 00:22:35.780054686 (2h30m51.560988072s) [noise=true , address=45.91.116.12:4226 , piece_id=Z7BOS5IU7OH3LJK5FR544QNNKBICQKUM6B7OLNUZKBFNT3VLHNVA, seg_pos={0 0}, upload=2316]: Uploading piece...
Jul 27 00:22:35.781269803 (2h30m51.562203189s) [noise=true , address=45.91.116.12:4227 , piece_id=VBWEJWG2HL5HASAM2O3SJ5BLO56EAVKKCZ7MUZZCP4WTHVKQQ47A, seg_pos={0 0}, upload=2316]: Uploading piece...
Jul 27 00:22:35.781796509 (2h30m51.562729898s) [noise=true , address=45.91.116.13:4155 , piece_id=DXQDYFKUQHC5SYOOT7MANPI5U6YZAF5AASTQWTZTWEWW4UYGCNEQ, seg_pos={0 0}, upload=2316]: Uploading piece...
Jul 27 00:22:35.782987131 (2h30m51.56392052s) [noise=true , address=77.90.31.10:4518 , piece_id=IAWAZEPPEVPCCNNXU5Y5RGWGMZBFBRZNHD22DAV5JUOKUIXW6VLQ, seg_pos={0 0}, upload=2316]: Uploading piece...
Jul 27 00:22:35.784188079 (2h30m51.565121467s) [noise=true , address=45.91.116.6:4973 , piece_id=CZRWSPQB4KHTOPRSJTSIIKABELTEHJUSXKO6OTPNPJJK2DVWKO5A, seg_pos={0 0}, upload=2316]: Uploading piece...
Jul 27 00:22:35.785505146 (2h30m51.566438526s) [noise=true , address=45.91.116.6:4974 , piece_id=Q3MDAYIJEVD4HMBOMJWJPVZRNN62AMPVOS7N3Z4C5CVM74CG6UTQ, seg_pos={0 0}, upload=2316]: Uploading piece...
Jul 27 00:22:35.786530557 (2h30m51.567463943s) [noise=true , address=45.91.116.9:4262 , piece_id=X6LWIEQ5NSGSKGJ3WGHQABIDCNDDLAEMZ7BKBYHHIPMEAKVRPVGA, seg_pos={0 0}, upload=2316]: Uploading piece...
Jul 27 00:22:35.787095609 (2h30m51.568029008s) [noise=true , address=45.91.116.5:4298 , piece_id=GSLSFIKGIYWEIAECQWBL6PSPT7BPKRK2SV4WSFKMHRX7C3Z4VAYA, seg_pos={0 0}, upload=2316]: Uploading piece...
Jul 27 00:22:35.868798021 (2h30m51.649731401s) [upload=2316]: Next returned nil segment
Jul 27 00:22:35.868817824 (2h30m51.649751208s) [upload=2316]: Waiting for error group managing segments...
Jul 27 00:22:35.873835790 (2h30m51.654769171s) [noise=true , address=45.91.116.5:4298 , piece_id=GSLSFIKGIYWEIAECQWBL6PSPT7BPKRK2SV4WSFKMHRX7C3Z4VAYA, seg_pos={0 0}, upload=2316]: Done uploading piece. err: <nil>
Jul 27 00:22:35.874532930 (2h30m51.655466307s) [noise=true , address=45.91.116.12:4227 , piece_id=VBWEJWG2HL5HASAM2O3SJ5BLO56EAVKKCZ7MUZZCP4WTHVKQQ47A, seg_pos={0 0}, upload=2316]: Done uploading piece. err: <nil>
Jul 27 00:22:35.875012943 (2h30m51.655946312s) [noise=true , address=45.91.116.11:4119 , piece_id=V4BNG3X6KOITLDNCFWT6XBDZOJCYDXO5H3N6AJ7MKKF4NXMP72AQ, seg_pos={0 0}, upload=2316]: Done uploading piece. err: <nil>
Jul 27 00:22:35.875470060 (2h30m51.65640343s) [noise=true , address=45.91.116.6:4974 , piece_id=Q3MDAYIJEVD4HMBOMJWJPVZRNN62AMPVOS7N3Z4C5CVM74CG6UTQ, seg_pos={0 0}, upload=2316]: Done uploading piece. err: <nil>
Jul 27 00:22:35.875854097 (2h30m51.656787466s) [noise=true , address=45.91.116.12:4226 , piece_id=Z7BOS5IU7OH3LJK5FR544QNNKBICQKUM6B7OLNUZKBFNT3VLHNVA, seg_pos={0 0}, upload=2316]: Done uploading piece. err: <nil>
Jul 27 00:22:35.876566577 (2h30m51.657499958s) [noise=true , address=45.91.116.13:4155 , piece_id=DXQDYFKUQHC5SYOOT7MANPI5U6YZAF5AASTQWTZTWEWW4UYGCNEQ, seg_pos={0 0}, upload=2316]: Done uploading piece. err: <nil>
Jul 27 00:22:35.877190659 (2h30m51.658124035s) [noise=true , address=45.91.116.10:4082 , piece_id=OA25WWKBZAQPR7NOM3PFIOR3L75WUAJ2WDSDFLTUOHWPEFQIXL3A, seg_pos={0 0}, upload=2316]: Done uploading piece. err: <nil>
Jul 27 00:22:35.877759205 (2h30m51.658692586s) [noise=true , address=45.91.116.6:4973 , piece_id=CZRWSPQB4KHTOPRSJTSIIKABELTEHJUSXKO6OTPNPJJK2DVWKO5A, seg_pos={0 0}, upload=2316]: Done uploading piece. err: <nil>
Jul 27 00:22:35.878468274 (2h30m51.659401638s) [noise=true , address=77.90.31.14:4663 , piece_id=S36NSRGS7IGRKBIOKHLZGMZLIKMXZRSCVZS5WJZIIP7FLGONKLHA, seg_pos={0 0}, upload=2316]: Done uploading piece. err: <nil>
Jul 27 00:22:35.904448561 (2h30m51.685381938s) [noise=true , address=45.91.116.9:4262 , piece_id=X6LWIEQ5NSGSKGJ3WGHQABIDCNDDLAEMZ7BKBYHHIPMEAKVRPVGA, seg_pos={0 0}, upload=2316]: Done uploading piece. err: <nil>
Jul 27 00:23:05.695637857 (2h31m21.476571232s) [noise=true , address=77.90.31.10:4517 , piece_id=4TEO72K4T5IBB3KAYDJ7XA3BMPSOIX4FGLOSM3AYBDVSKRVPGX7A, seg_pos={0 0}, upload=2316]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: rpc: dial tcp 77.90.31.10:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:23:05.695721000 (2h31m21.476654377s) [noise=true , address=104.36.82.14:4223 , piece_id=364455LQ6FYYIA6II4JS7PKFZM7422U2B2KD4DD3VGYVJPQ2YZWA, seg_pos={0 0}, upload=2316]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:23:05.695783323 (2h31m21.476716699s) [noise=true , address=77.90.31.10:4518 , piece_id=IAWAZEPPEVPCCNNXU5Y5RGWGMZBFBRZNHD22DAV5JUOKUIXW6VLQ, seg_pos={0 0}, upload=2316]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: rpc: dial tcp 77.90.31.10:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:23:05.695822693 (2h31m21.476756068s) [noise=true , address=104.36.82.16:4295 , piece_id=3CNUVGW4XWZB4PMYKNWVKHKNRQH5M4XYWLULZ6HPY4GYBJBUYWIQ, seg_pos={0 0}, upload=2316]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:23:05.695836853 (2h31m21.476770228s) [noise=true , address=104.36.82.2:4731 , piece_id=TAHBPPGVULUOM3KNMPUWBGLHUJMETCHHBPWZSP5VRFG3GSTEZSGA, seg_pos={0 0}, upload=2316]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:23:05.695874418 (2h31m21.476807783s) [noise=true , address=77.90.31.21:4878 , piece_id=27RVSTANPTACCVI7UO2LIM2K46SIGQX4A47X7AHKZP2KQPK5XHUA, seg_pos={0 0}, upload=2316]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:42704->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:23:05.695985434 (2h31m21.476918815s) [seg_pos={0 0}, upload=2316]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 27 00:23:05.719386464 (2h31m21.500319826s) [upload=2316]: Done uploading segments...
Jul 27 00:23:05.719418120 (2h31m21.500351499s) [upload=2316]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 30.02618536s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [...........X.X........X...XX...............................X] duration:
1m0.002104184s
FAIL (2025-07-27 00:22:56 UTC)
operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.002152392s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [.XXX...............X..X......X.XX....X...X.................X] duration:
30.000767384s
FAIL (2025-07-27 00:22:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.0008631s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
40 out of 60 is failed (threshold=3) [XXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.X] duration:
30.027159506s
FAIL (2025-07-27 00:22:32 UTC)
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 6)
Duration: 30.027191673s
core
uplink
us1
vivint
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.032764142s
FAIL (2025-07-27 00:22:25 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
BatchRequestItem_SegmentBegin
Jul 27 00:21:25.845611332 (2h29m41.626544712s) [seg_pos={0 0}, upload=2299]: Begin upload segment...
Jul 27 00:21:25.845681099 (2h29m41.626614467s) [seg_pos={0 0}, upload=2299]: Done begin upload segment.
Jul 27 00:21:25.845710878 (2h29m41.626644259s) [noise=true , address=79.127.205.234:4607 , piece_id=B54J4QSM5AE5GDEKOKBO34G5ZDXKN2SKQU6NG5VYOV62TGMMSZIA, seg_pos={0 0}, upload=2299]: Uploading piece...
Jul 27 00:21:25.845740303 (2h29m41.626673688s) [noise=true , address=77.90.31.16:4759 , piece_id=5NR4O3SPBDGEPESQ7CMTO5EMB3FINRPHSHBPKG2HVLWDJOUEDPMQ, seg_pos={0 0}, upload=2299]: Uploading piece...
Jul 27 00:21:25.846239235 (2h29m41.627172615s) [noise=true , address=79.127.201.215:4346 , piece_id=JWCVVLU6WF6DIFSO7QZ3MJVV3E55PXSW7KLWYA5Y7R5WU55VC4XQ, seg_pos={0 0}, upload=2299]: Uploading piece...
Jul 27 00:21:25.846317932 (2h29m41.627251318s) [noise=true , address=79.127.201.214:4289 , piece_id=BE3AO3IK2A4EITAPW2AQ3ZI57OA3PXMGAKTQ2FIFAK6VKLOHXP2A, seg_pos={0 0}, upload=2299]: Uploading piece...
Jul 27 00:21:25.846656216 (2h29m41.627589623s) [noise=true , address=104.36.82.55:4201 , piece_id=AFLD4XESUM7DQARC7URD62VSUK5XBNWOQ2VBAAVHQ72GSSD3SNVA, seg_pos={0 0}, upload=2299]: Uploading piece...
Jul 27 00:21:25.846842935 (2h29m41.627776317s) [noise=true , address=104.36.82.56:4260 , piece_id=2BV2FFWULCV6UVYYI5G3RNWTGLWCKNAFLHWW7VPRQVBRNXKMDSFQ, seg_pos={0 0}, upload=2299]: Uploading piece...
Jul 27 00:21:25.847082090 (2h29m41.628015504s) [noise=true , address=79.127.205.233:4655 , piece_id=B6R6YUL37ETSDZRW5WWVIP2NMAFDRHFZ6MENEUEQ3JOIV26LTEIQ, seg_pos={0 0}, upload=2299]: Uploading piece...
Jul 27 00:21:25.847389221 (2h29m41.628322601s) [noise=true , address=77.90.31.10:4531 , piece_id=XUC6OOB3CNEG7JRLVHLJ7EK64NDVDCL723XIAI7SHX76P6JCBUFQ, seg_pos={0 0}, upload=2299]: Uploading piece...
Jul 27 00:21:25.847529182 (2h29m41.628462569s) [noise=true , address=156.146.43.225:4448 , piece_id=Z6ZQXQR3GLMXQKYS3RHNTCCGJXFLUW2GB5BZM6VHVHKJFD46MQMA, seg_pos={0 0}, upload=2299]: Uploading piece...
Jul 27 00:21:25.847864818 (2h29m41.628798207s) [noise=true , address=104.36.82.4:4832 , piece_id=LQXHJXDCFOH5VOJUDU5HZBHJLXW467LCBR55C4ZDARJIOR672ZQQ, seg_pos={0 0}, upload=2299]: Uploading piece...
Jul 27 00:21:25.848089684 (2h29m41.629023066s) [noise=true , address=64.49.13.140:4508 , piece_id=MB3PMTNGBQJLEORLRX5BE4DJBEUXSUGU5D4H63JGFJGXBSZYUOOA, seg_pos={0 0}, upload=2299]: Uploading piece...
Jul 27 00:21:25.848411645 (2h29m41.629345029s) [noise=true , address=77.90.31.13:4656 , piece_id=R4OVOCMZ6FPZRMSVTPLHXKPXY5DSSKFPTBJFQL4PRXG37AIPG62A, seg_pos={0 0}, upload=2299]: Uploading piece...
Jul 27 00:21:25.848460712 (2h29m41.6293941s) [noise=true , address=64.49.13.138:4483 , piece_id=YQTUONU3E3SF5ZTFD7RKZEGWOPVIWTKYK3UD3O3ONNFKETA6L2DA, seg_pos={0 0}, upload=2299]: Uploading piece...
Jul 27 00:21:25.848886460 (2h29m41.629819842s) [noise=true , address=64.49.13.26:4953 , piece_id=5HFKVX5V7KHR7Z7MMIHQ2DJHBDOHU6ONEZHKZMZKKQ7XLCRNES5A, seg_pos={0 0}, upload=2299]: Uploading piece...
Jul 27 00:21:25.849374433 (2h29m41.630307817s) [noise=true , address=104.36.82.47:4543 , piece_id=SI4VNAMTDUZ4WM33M5HAYYZ2QJATNWOBEBEI3WVOLMRH42E4BUOQ, seg_pos={0 0}, upload=2299]: Uploading piece...
Jul 27 00:21:25.849853476 (2h29m41.630786858s) [noise=true , address=79.127.205.246:4276 , piece_id=3L5UP6CPVF23R37T6XIXVY4SLZRUSCMBGVCIOTQRR7WM6NN44R6Q, seg_pos={0 0}, upload=2299]: Uploading piece...
Jul 27 00:21:25.850498602 (2h29m41.631431984s) [noise=true , address=104.36.82.52:4110 , piece_id=OCYBKVDGL25E6UPEIEKP5YNC525MRFHWUOPZZUVFBJHY37663ZUQ, seg_pos={0 0}, upload=2299]: Uploading piece...
Jul 27 00:21:25.850688866 (2h29m41.631622302s) [noise=true , address=77.90.31.11:4574 , piece_id=TJ7NTCZF4PUH5CUOTVK56DCSTYGFMAKNYZ5D7MHEFTEJLSWM5MBA, seg_pos={0 0}, upload=2299]: Uploading piece...
Jul 27 00:21:25.950770091 (2h29m41.731703477s) [upload=2299]: Next returned nil segment
Jul 27 00:21:25.950794342 (2h29m41.731727725s) [upload=2299]: Waiting for error group managing segments...
Jul 27 00:21:26.560293150 (2h29m42.341226547s) [noise=true , address=79.127.205.246:4276 , piece_id=3L5UP6CPVF23R37T6XIXVY4SLZRUSCMBGVCIOTQRR7WM6NN44R6Q, seg_pos={0 0}, upload=2299]: Done uploading piece. err: <nil>
Jul 27 00:21:26.692037765 (2h29m42.472971148s) [noise=true , address=79.127.205.233:4655 , piece_id=B6R6YUL37ETSDZRW5WWVIP2NMAFDRHFZ6MENEUEQ3JOIV26LTEIQ, seg_pos={0 0}, upload=2299]: Done uploading piece. err: <nil>
Jul 27 00:21:26.746683600 (2h29m42.527616979s) [noise=true , address=77.90.31.16:4759 , piece_id=5NR4O3SPBDGEPESQ7CMTO5EMB3FINRPHSHBPKG2HVLWDJOUEDPMQ, seg_pos={0 0}, upload=2299]: Done uploading piece. err: <nil>
Jul 27 00:21:26.856216536 (2h29m42.637149905s) [noise=true , address=79.127.205.234:4607 , piece_id=B54J4QSM5AE5GDEKOKBO34G5ZDXKN2SKQU6NG5VYOV62TGMMSZIA, seg_pos={0 0}, upload=2299]: Done uploading piece. err: <nil>
Jul 27 00:21:28.719383028 (2h29m44.500316408s) [noise=true , address=156.146.43.225:4448 , piece_id=Z6ZQXQR3GLMXQKYS3RHNTCCGJXFLUW2GB5BZM6VHVHKJFD46MQMA, seg_pos={0 0}, upload=2299]: Done uploading piece. err: <nil>
Jul 27 00:21:28.771688449 (2h29m44.552621814s) [noise=true , address=64.49.13.26:4953 , piece_id=5HFKVX5V7KHR7Z7MMIHQ2DJHBDOHU6ONEZHKZMZKKQ7XLCRNES5A, seg_pos={0 0}, upload=2299]: Done uploading piece. err: <nil>
Jul 27 00:21:30.552322605 (2h29m46.333255993s) [noise=true , address=79.127.201.214:4289 , piece_id=BE3AO3IK2A4EITAPW2AQ3ZI57OA3PXMGAKTQ2FIFAK6VKLOHXP2A, seg_pos={0 0}, upload=2299]: Done uploading piece. err: <nil>
Jul 27 00:21:32.747795444 (2h29m48.528728808s) [noise=true , address=64.49.13.140:4508 , piece_id=MB3PMTNGBQJLEORLRX5BE4DJBEUXSUGU5D4H63JGFJGXBSZYUOOA, seg_pos={0 0}, upload=2299]: Done uploading piece. err: <nil>
Jul 27 00:21:38.044431639 (2h29m53.825365025s) [noise=true , address=64.49.13.138:4483 , piece_id=YQTUONU3E3SF5ZTFD7RKZEGWOPVIWTKYK3UD3O3ONNFKETA6L2DA, seg_pos={0 0}, upload=2299]: Done uploading piece. err: <nil>
Jul 27 00:21:48.213401071 (2h30m3.994334455s) [noise=true , address=79.127.201.215:4346 , piece_id=JWCVVLU6WF6DIFSO7QZ3MJVV3E55PXSW7KLWYA5Y7R5WU55VC4XQ, seg_pos={0 0}, upload=2299]: Done uploading piece. err: <nil>
Jul 27 00:22:11.958712268 (2h30m27.739645638s) [noise=true , address=77.90.31.13:4656 , piece_id=R4OVOCMZ6FPZRMSVTPLHXKPXY5DSSKFPTBJFQL4PRXG37AIPG62A, seg_pos={0 0}, upload=2299]: Done uploading piece. err: <nil>
Jul 27 00:22:25.772740938 (2h30m41.553674306s) [noise=true , address=104.36.82.52:4110 , piece_id=OCYBKVDGL25E6UPEIEKP5YNC525MRFHWUOPZZUVFBJHY37663ZUQ, seg_pos={0 0}, upload=2299]: Done uploading piece. err: ecclient: upload failed (node:1ftGsmDR7RLZnGZzHLnE7g4PSTkpBo4JLN5MBaXPjPciPETYui, address:104.36.82.52:4110): protocol: rpc: dial tcp 104.36.82.52:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:22:25.772816418 (2h30m41.553749784s) [noise=true , address=104.36.82.55:4201 , piece_id=AFLD4XESUM7DQARC7URD62VSUK5XBNWOQ2VBAAVHQ72GSSD3SNVA, seg_pos={0 0}, upload=2299]: Done uploading piece. err: ecclient: upload failed (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi, address:104.36.82.55:4201): protocol: rpc: dial tcp 104.36.82.55:4201: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:22:25.772885980 (2h30m41.553819347s) [noise=true , address=104.36.82.47:4543 , piece_id=SI4VNAMTDUZ4WM33M5HAYYZ2QJATNWOBEBEI3WVOLMRH42E4BUOQ, seg_pos={0 0}, upload=2299]: Done uploading piece. err: ecclient: upload failed (node:12KxnBTVxhbwA79Gd8V7Mh6bWybtqttqgo5agHoHewXHAey5y7D, address:104.36.82.47:4543): protocol: write tcp 159.223.143.39:50240->104.36.82.47:4543: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:22:25.773084199 (2h30m41.554017561s) [noise=true , address=104.36.82.56:4260 , piece_id=2BV2FFWULCV6UVYYI5G3RNWTGLWCKNAFLHWW7VPRQVBRNXKMDSFQ, seg_pos={0 0}, upload=2299]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: rpc: dial tcp 104.36.82.56:4260: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:22:25.773110483 (2h30m41.554043868s) [noise=true , address=77.90.31.10:4531 , piece_id=XUC6OOB3CNEG7JRLVHLJ7EK64NDVDCL723XIAI7SHX76P6JCBUFQ, seg_pos={0 0}, upload=2299]: Done uploading piece. err: ecclient: upload failed (node:1AvSrhj1nHWLBJrw1SKacnjPTi9g4AzjqKvVFoxtzAdJFYAfQt, address:77.90.31.10:4531): protocol: rpc: dial tcp 77.90.31.10:4531: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:22:25.773216295 (2h30m41.554149663s) [noise=true , address=104.36.82.4:4832 , piece_id=LQXHJXDCFOH5VOJUDU5HZBHJLXW467LCBR55C4ZDARJIOR672ZQQ, seg_pos={0 0}, upload=2299]: Done uploading piece. err: ecclient: upload failed (node:1SWdNMwLYv7KiBqr5Pfo8qvxHnxirHGUDbhZWeojGXwueQUqr9, address:104.36.82.4:4832): protocol: write tcp 159.223.143.39:52510->104.36.82.4:4832: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:22:25.773249204 (2h30m41.554182587s) [noise=true , address=77.90.31.11:4574 , piece_id=TJ7NTCZF4PUH5CUOTVK56DCSTYGFMAKNYZ5D7MHEFTEJLSWM5MBA, seg_pos={0 0}, upload=2299]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: rpc: dial tcp 77.90.31.11:4574: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:22:25.773313587 (2h30m41.554246972s) [seg_pos={0 0}, upload=2299]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 27 00:22:25.802897449 (2h30m41.583830828s) [upload=2299]: Done uploading segments...
Jul 27 00:22:25.802931970 (2h30m41.583865347s) [upload=2299]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 1m0.032787184s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
43 out of 60 is failed (threshold=3) [XXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XX] duration:
30.027886622s
FAIL (2025-07-27 00:22:07 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: rpc: dial tcp 77.90.31.11:4553: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:20:57.006580930 (2h29m12.787514304s) [noise=true , address=77.90.31.21:4877 , piece_id=67JFI32F7BX2ZNADJPNMT4XWE3P6IRNYNWEKZV3VUF5KQRAPP2MQ, seg_pos={0 0}, upload=2287]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded
Jul 27 00:20:57.006650182 (2h29m12.787583572s) [seg_pos={0 0}, upload=2287]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 27 00:20:57.034251290 (2h29m12.815184674s) [upload=2287]: Done uploading segments...
Jul 27 00:20:57.034293667 (2h29m12.815227031s) [upload=2287]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Jul 27 00:21:37.028547529 (2h29m52.809480919s) [upload=2302]: Starting upload
Jul 27 00:21:37.028652520 (2h29m52.809585908s) [upload=2302]: Done starting upload
Jul 27 00:21:37.028754843 (2h29m52.809688226s) [upload=2302]: Uploading segments...
Jul 27 00:21:37.029190281 (2h29m52.81012367s) [upload=2302]: Got next segment. Inline: false
Jul 27 00:21:37.029248393 (2h29m52.810181785s) [upload=2302]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 27 00:21:37.029256592 (2h29m52.810189978s) [upload=2302]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 27 00:21:37.060493438 (2h29m52.841426836s) [seg_pos={0 0}, upload=2302]: Begin upload segment...
Jul 27 00:21:37.060553105 (2h29m52.841486487s) [seg_pos={0 0}, upload=2302]: Done begin upload segment.
Jul 27 00:21:37.060585770 (2h29m52.841519158s) [noise=true , address=104.36.82.14:4223 , piece_id=NYJ2SS7HIAJCBUI5D4LWUA5QSTDBO6QDNEPYAUEUCHGCJNJRJLFQ, seg_pos={0 0}, upload=2302]: Uploading piece...
Jul 27 00:21:37.061114396 (2h29m52.842047781s) [noise=true , address=45.91.116.11:4117 , piece_id=AG5TRPYVVSW4OIOKNBGWJ4YGA5KTHZV4PZBOCG4MESFULBX4VPZQ, seg_pos={0 0}, upload=2302]: Uploading piece...
Jul 27 00:21:37.061522230 (2h29m52.84245562s) [noise=true , address=104.36.82.16:4295 , piece_id=JMN6GMFWIATKVDZYJOFEDHQS4JUPW7ZIW7C4N6RKU7UUXD7CU3IQ, seg_pos={0 0}, upload=2302]: Uploading piece...
Jul 27 00:21:37.061911594 (2h29m52.842844983s) [noise=true , address=45.91.116.10:4083 , piece_id=FMBUNIBVIYKSKXHX5R2QGRN224ISWS4KXGN7EKEZ2PUIZISWFEKA, seg_pos={0 0}, upload=2302]: Uploading piece...
Jul 27 00:21:37.062297035 (2h29m52.843230418s) [noise=true , address=45.91.116.8:4045 , piece_id=VEKBCCJBSXNK5ZRV4UBJ2Z2X6X6JP5D7B7TB7PCG477OMTC7ZNEQ, seg_pos={0 0}, upload=2302]: Uploading piece...
Jul 27 00:21:37.062687955 (2h29m52.843621339s) [noise=true , address=45.91.116.13:4154 , piece_id=DVEQGDU36AC3TL4WX77R7BNPJ52QWH5PHK6YFGTWCOAPWOMLU3KA, seg_pos={0 0}, upload=2302]: Uploading piece...
Jul 27 00:21:37.063098303 (2h29m52.844031691s) [noise=true , address=45.91.116.12:4225 , piece_id=SPUQTI5EMC7DM3DEUTTC7X4ZOORKHSMKYMD67P4SORRJ5TWO3TDQ, seg_pos={0 0}, upload=2302]: Uploading piece...
Jul 27 00:21:37.063483303 (2h29m52.844416687s) [noise=true , address=45.91.116.4:4937 , piece_id=6MM2OUY57MADCYUBSODXLBZ6Q62NIPR2GY2XYQIBYJVHPLKNEBMA, seg_pos={0 0}, upload=2302]: Uploading piece...
Jul 27 00:21:37.063952392 (2h29m52.844885788s) [noise=true , address=104.36.82.8:4971 , piece_id=KSKXDPTXCJMZKL3JBL5QRNQIJ2X7ZLULFU62CQPP6LNRT5KBUJSA, seg_pos={0 0}, upload=2302]: Uploading piece...
Jul 27 00:21:37.064322978 (2h29m52.845256363s) [noise=true , address=104.36.82.6:4873 , piece_id=EURLWODJCUALWBRTOLZLY646L7W2TDEOB45YVJJY66RHP5OH4MRA, seg_pos={0 0}, upload=2302]: Uploading piece...
Jul 27 00:21:37.064665253 (2h29m52.845598618s) [noise=true , address=45.91.116.14:4191 , piece_id=OBZWTD6J4ZWY252WY6LHK3RBEO3WKPQGHDN2MWDDIRVLWA5RHKFQ, seg_pos={0 0}, upload=2302]: Uploading piece...
Jul 27 00:21:37.065010995 (2h29m52.845944379s) [noise=true , address=77.90.31.14:4661 , piece_id=KCPPL3E27PGFJRZLDQ5V3BWPYTRAERHQLWIDX6KOABOS4NBR2JWQ, seg_pos={0 0}, upload=2302]: Uploading piece...
Jul 27 00:21:37.065416498 (2h29m52.846349899s) [noise=true , address=45.91.116.9:4261 , piece_id=BO7BHA3ZWYI7UNFMJG6R3OCEZFKKRDZP3FVCXR3FAIDWF34DZRMA, seg_pos={0 0}, upload=2302]: Uploading piece...
Jul 27 00:21:37.065784904 (2h29m52.84671829s) [noise=true , address=45.91.116.7:4009 , piece_id=JQHUZYFLLL2SHUNVBC2MLYBUK6OVJYA3R73OC5HINV4C7TDVKJCA, seg_pos={0 0}, upload=2302]: Uploading piece...
Jul 27 00:21:37.066256381 (2h29m52.847189881s) [noise=true , address=45.91.116.6:4974 , piece_id=ZZOAL4EUDFJ7WOAPAIDGP24FFNB25VG7K3W7365LYROVG3MGHXCQ, seg_pos={0 0}, upload=2302]: Uploading piece...
Jul 27 00:21:37.066660270 (2h29m52.847593656s) [noise=true , address=77.90.31.16:4735 , piece_id=RZJRH6QHIEE6CMIX32FUIOPYLKANZH54JJLOXSPUEH7Z4LJXY4NQ, seg_pos={0 0}, upload=2302]: Uploading piece...
Jul 27 00:21:37.167427333 (2h29m52.948360715s) [upload=2302]: Next returned nil segment
Jul 27 00:21:37.167453250 (2h29m52.948386644s) [upload=2302]: Waiting for error group managing segments...
Jul 27 00:21:37.171892029 (2h29m52.952825407s) [noise=true , address=45.91.116.13:4154 , piece_id=DVEQGDU36AC3TL4WX77R7BNPJ52QWH5PHK6YFGTWCOAPWOMLU3KA, seg_pos={0 0}, upload=2302]: Done uploading piece. err: <nil>
Jul 27 00:21:37.172684508 (2h29m52.953617888s) [noise=true , address=45.91.116.12:4225 , piece_id=SPUQTI5EMC7DM3DEUTTC7X4ZOORKHSMKYMD67P4SORRJ5TWO3TDQ, seg_pos={0 0}, upload=2302]: Done uploading piece. err: <nil>
Jul 27 00:21:37.173263631 (2h29m52.954196996s) [noise=true , address=45.91.116.7:4009 , piece_id=JQHUZYFLLL2SHUNVBC2MLYBUK6OVJYA3R73OC5HINV4C7TDVKJCA, seg_pos={0 0}, upload=2302]: Done uploading piece. err: <nil>
Jul 27 00:21:37.173867164 (2h29m52.954800536s) [noise=true , address=77.90.31.14:4661 , piece_id=KCPPL3E27PGFJRZLDQ5V3BWPYTRAERHQLWIDX6KOABOS4NBR2JWQ, seg_pos={0 0}, upload=2302]: Done uploading piece. err: <nil>
Jul 27 00:21:37.174384305 (2h29m52.955317673s) [noise=true , address=45.91.116.9:4261 , piece_id=BO7BHA3ZWYI7UNFMJG6R3OCEZFKKRDZP3FVCXR3FAIDWF34DZRMA, seg_pos={0 0}, upload=2302]: Done uploading piece. err: <nil>
Jul 27 00:21:37.174885307 (2h29m52.955818684s) [noise=true , address=45.91.116.4:4937 , piece_id=6MM2OUY57MADCYUBSODXLBZ6Q62NIPR2GY2XYQIBYJVHPLKNEBMA, seg_pos={0 0}, upload=2302]: Done uploading piece. err: <nil>
Jul 27 00:21:37.175399043 (2h29m52.956332424s) [noise=true , address=45.91.116.8:4045 , piece_id=VEKBCCJBSXNK5ZRV4UBJ2Z2X6X6JP5D7B7TB7PCG477OMTC7ZNEQ, seg_pos={0 0}, upload=2302]: Done uploading piece. err: <nil>
Jul 27 00:21:37.175908678 (2h29m52.956842045s) [noise=true , address=45.91.116.11:4117 , piece_id=AG5TRPYVVSW4OIOKNBGWJ4YGA5KTHZV4PZBOCG4MESFULBX4VPZQ, seg_pos={0 0}, upload=2302]: Done uploading piece. err: <nil>
Jul 27 00:21:37.176528987 (2h29m52.957462351s) [noise=true , address=45.91.116.10:4083 , piece_id=FMBUNIBVIYKSKXHX5R2QGRN224ISWS4KXGN7EKEZ2PUIZISWFEKA, seg_pos={0 0}, upload=2302]: Done uploading piece. err: <nil>
Jul 27 00:21:37.176825269 (2h29m52.957758638s) [noise=true , address=45.91.116.6:4974 , piece_id=ZZOAL4EUDFJ7WOAPAIDGP24FFNB25VG7K3W7365LYROVG3MGHXCQ, seg_pos={0 0}, upload=2302]: Done uploading piece. err: <nil>
Jul 27 00:21:37.212768672 (2h29m52.993702053s) [noise=true , address=45.91.116.14:4191 , piece_id=OBZWTD6J4ZWY252WY6LHK3RBEO3WKPQGHDN2MWDDIRVLWA5RHKFQ, seg_pos={0 0}, upload=2302]: Done uploading piece. err: <nil>
Jul 27 00:21:38.615196614 (2h29m54.396129978s) [noise=true , address=77.90.31.16:4735 , piece_id=RZJRH6QHIEE6CMIX32FUIOPYLKANZH54JJLOXSPUEH7Z4LJXY4NQ, seg_pos={0 0}, upload=2302]: Done uploading piece. err: <nil>
Jul 27 00:22:06.992686366 (2h30m22.773619762s) [noise=true , address=104.36.82.8:4971 , piece_id=KSKXDPTXCJMZKL3JBL5QRNQIJ2X7ZLULFU62CQPP6LNRT5KBUJSA, seg_pos={0 0}, upload=2302]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 159.223.143.39:44652->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:22:06.992787636 (2h30m22.773721013s) [noise=true , address=104.36.82.16:4295 , piece_id=JMN6GMFWIATKVDZYJOFEDHQS4JUPW7ZIW7C4N6RKU7UUXD7CU3IQ, seg_pos={0 0}, upload=2302]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:22:06.992854441 (2h30m22.773787819s) [noise=true , address=104.36.82.6:4873 , piece_id=EURLWODJCUALWBRTOLZLY646L7W2TDEOB45YVJJY66RHP5OH4MRA, seg_pos={0 0}, upload=2302]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 159.223.143.39:48040->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:22:06.992919905 (2h30m22.773853268s) [noise=true , address=104.36.82.14:4223 , piece_id=NYJ2SS7HIAJCBUI5D4LWUA5QSTDBO6QDNEPYAUEUCHGCJNJRJLFQ, seg_pos={0 0}, upload=2302]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:58496->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:22:06.992951239 (2h30m22.773884626s) [seg_pos={0 0}, upload=2302]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 27 00:22:07.017432150 (2h30m22.798365536s) [upload=2302]: Done uploading segments...
Jul 27 00:22:07.017466365 (2h30m22.79839975s) [upload=2302]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.027913004s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [............X.X........X...XX...............................] duration:
41.358878973s
FAIL (2025-07-27 00:21:27 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 41.358896966s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
1m0.02982462s
FAIL (2025-07-27 00:21:25 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
oading piece...
Jul 27 00:20:25.750008228 (2h28m41.530941609s) [noise=true , address=104.36.82.52:4103 , piece_id=7XF3HHNNUEUM2XT7U72JN4ELTUSM4IJ7AQTWNKVOPBDAO7NG3VZA, seg_pos={0 0}, upload=2286]: Uploading piece...
Jul 27 00:20:25.750398065 (2h28m41.531331448s) [noise=true , address=79.127.205.245:4244 , piece_id=NZPKZPY6DRVJCLQEHMSLQGCGVJMSH4X75KQPQFU6522N3DX4KJTQ, seg_pos={0 0}, upload=2286]: Uploading piece...
Jul 27 00:20:25.750413229 (2h28m41.531346617s) [noise=true , address=104.36.82.15:4270 , piece_id=UXNWIMSPDEYXQCC6GVZRBNHFHLEYNIBCEFMUQFZ3IEEZIDFPPOFA, seg_pos={0 0}, upload=2286]: Uploading piece...
Jul 27 00:20:25.750942148 (2h28m41.531875532s) [noise=true , address=104.36.82.44:4874 , piece_id=YA7TS3PFJUES3YOS3S4PJTV6IO6TRTSNRUXNQ3JNA6Y22LDASQRA, seg_pos={0 0}, upload=2286]: Uploading piece...
Jul 27 00:20:25.751838164 (2h28m41.532771546s) [noise=true , address=79.127.205.234:4613 , piece_id=AUPGDA24IXGQCY7BJPBHGT3IKHUGN5QKQRTL2DUIBK2IGDUBIJPA, seg_pos={0 0}, upload=2286]: Uploading piece...
Jul 27 00:20:25.752463952 (2h28m41.53339733s) [noise=true , address=79.127.205.235:4685 , piece_id=XTQG6BM3UDR5GPOFDBSJ754ZFV6QUMZFGIJWN6UJYCOJSGPVGRSQ, seg_pos={0 0}, upload=2286]: Uploading piece...
Jul 27 00:20:25.752947463 (2h28m41.533880849s) [noise=true , address=79.127.201.215:4346 , piece_id=V4LS32XF67KJETKZDNWEZQTBHGFGJUAN3CRCCJLMW7XMS2ALOVZQ, seg_pos={0 0}, upload=2286]: Uploading piece...
Jul 27 00:20:25.753452630 (2h28m41.534386019s) [noise=true , address=104.36.82.54:4183 , piece_id=J6HARDQZWX77R3HN3W435AFD3HLSWQO4IADDFXPYL5V6GAOVYJTQ, seg_pos={0 0}, upload=2286]: Uploading piece...
Jul 27 00:20:25.753885872 (2h28m41.534819258s) [noise=true , address=104.36.82.47:4548 , piece_id=4XZC5P4LMAMYUGCXDV53VPPSJE2LHWHDPGVLB2YLPDWD2UR6JWVA, seg_pos={0 0}, upload=2286]: Uploading piece...
Jul 27 00:20:25.754327743 (2h28m41.53526113s) [noise=true , address=77.90.31.16:4751 , piece_id=KQVRGH5PABRYJEM2ZCINO6WDOTWYDKB3AH3VB765MAWN7XVBIBWA, seg_pos={0 0}, upload=2286]: Uploading piece...
Jul 27 00:20:25.754848598 (2h28m41.535781983s) [noise=true , address=104.36.82.56:4260 , piece_id=AASN7ZRQJBVEFBL5OQSVGRXXNN52IJZZ4WMJMFFX6QKM6G3R7MBQ, seg_pos={0 0}, upload=2286]: Uploading piece...
Jul 27 00:20:25.755341568 (2h28m41.53627495s) [noise=true , address=77.90.31.19:4847 , piece_id=WZ6J5BIHGKJTVBKFUHNS7Q6NGIMJMA6WKPVQUKCGB36MUCRIK4TQ, seg_pos={0 0}, upload=2286]: Uploading piece...
Jul 27 00:20:25.755893713 (2h28m41.536827096s) [noise=true , address=104.36.82.46:4509 , piece_id=5L6LGGFXDTYNYIPB7ZXZAWXZHC35WEVBOOELVOGD4YFBQS2KMQOQ, seg_pos={0 0}, upload=2286]: Uploading piece...
Jul 27 00:20:25.756309800 (2h28m41.537243184s) [noise=true , address=77.90.31.11:4574 , piece_id=WQUYLBLMJSEJBUN6OYGXH2WYJ3T2ANBTZEHNWMPHH2ZXCCG4SKTQ, seg_pos={0 0}, upload=2286]: Uploading piece...
Jul 27 00:20:25.756677341 (2h28m41.537610726s) [noise=true , address=156.146.43.225:4448 , piece_id=2HGVZSO4GOKDJOVHXE2MN4MAMW26WLA6ZM6LWN4EBGC6IHI2UEXA, seg_pos={0 0}, upload=2286]: Uploading piece...
Jul 27 00:20:25.833891044 (2h28m41.614824416s) [upload=2286]: Next returned nil segment
Jul 27 00:20:25.833914457 (2h28m41.614847853s) [upload=2286]: Waiting for error group managing segments...
Jul 27 00:20:26.124753455 (2h28m41.905686842s) [noise=true , address=64.49.13.140:4514 , piece_id=QUAVPSXBHZXPI5WMUXPIVIHVVN7J24SA4TSPADVV4YONM7KKFTUQ, seg_pos={0 0}, upload=2286]: Done uploading piece. err: ecclient: upload failed (node:1WsbDX2Zy8FYnEQtUsYFHSeDHj1DkkGyWBPaxzmReWFzaTejHn, address:64.49.13.140:4514): protocol: storage node overloaded, request limit: 2000
Jul 27 00:20:26.361148451 (2h28m42.142081829s) [noise=true , address=79.127.205.234:4613 , piece_id=AUPGDA24IXGQCY7BJPBHGT3IKHUGN5QKQRTL2DUIBK2IGDUBIJPA, seg_pos={0 0}, upload=2286]: Done uploading piece. err: <nil>
Jul 27 00:20:26.406600519 (2h28m42.187533909s) [noise=true , address=79.127.205.235:4685 , piece_id=XTQG6BM3UDR5GPOFDBSJ754ZFV6QUMZFGIJWN6UJYCOJSGPVGRSQ, seg_pos={0 0}, upload=2286]: Done uploading piece. err: <nil>
Jul 27 00:20:26.609224146 (2h28m42.390157526s) [noise=true , address=79.127.205.245:4244 , piece_id=NZPKZPY6DRVJCLQEHMSLQGCGVJMSH4X75KQPQFU6522N3DX4KJTQ, seg_pos={0 0}, upload=2286]: Done uploading piece. err: <nil>
Jul 27 00:20:26.678160553 (2h28m42.45909392s) [noise=true , address=79.127.201.215:4346 , piece_id=V4LS32XF67KJETKZDNWEZQTBHGFGJUAN3CRCCJLMW7XMS2ALOVZQ, seg_pos={0 0}, upload=2286]: Done uploading piece. err: <nil>
Jul 27 00:20:27.166649857 (2h28m42.947583239s) [noise=true , address=156.146.43.225:4448 , piece_id=2HGVZSO4GOKDJOVHXE2MN4MAMW26WLA6ZM6LWN4EBGC6IHI2UEXA, seg_pos={0 0}, upload=2286]: Done uploading piece. err: <nil>
Jul 27 00:20:29.922900751 (2h28m45.70383412s) [noise=true , address=77.90.31.16:4751 , piece_id=KQVRGH5PABRYJEM2ZCINO6WDOTWYDKB3AH3VB765MAWN7XVBIBWA, seg_pos={0 0}, upload=2286]: Done uploading piece. err: <nil>
Jul 27 00:20:38.820722829 (2h28m54.601656199s) [noise=true , address=77.90.31.19:4847 , piece_id=WZ6J5BIHGKJTVBKFUHNS7Q6NGIMJMA6WKPVQUKCGB36MUCRIK4TQ, seg_pos={0 0}, upload=2286]: Done uploading piece. err: <nil>
Jul 27 00:21:25.681480878 (2h29m41.462414257s) [noise=true , address=104.36.82.56:4260 , piece_id=AASN7ZRQJBVEFBL5OQSVGRXXNN52IJZZ4WMJMFFX6QKM6G3R7MBQ, seg_pos={0 0}, upload=2286]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: rpc: dial tcp 104.36.82.56:4260: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:21:25.681572325 (2h29m41.462505703s) [noise=true , address=104.36.82.46:4509 , piece_id=5L6LGGFXDTYNYIPB7ZXZAWXZHC35WEVBOOELVOGD4YFBQS2KMQOQ, seg_pos={0 0}, upload=2286]: Done uploading piece. err: ecclient: upload failed (node:1y2NmauNvCyrMJ5z66Lihb3h3C9nwdJYqfSfNmC5dwDGYQqUKh, address:104.36.82.46:4509): protocol: rpc: dial tcp 104.36.82.46:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:21:25.681646155 (2h29m41.462579533s) [noise=true , address=104.36.82.55:4201 , piece_id=H43645BI3ZAW6HOOGO4YINQMNW2KXW7RANSQVV2WGW7765IBSQUA, seg_pos={0 0}, upload=2286]: Done uploading piece. err: ecclient: upload failed (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi, address:104.36.82.55:4201): protocol: rpc: dial tcp 104.36.82.55:4201: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:21:25.681703115 (2h29m41.462636493s) [noise=true , address=104.36.82.54:4183 , piece_id=J6HARDQZWX77R3HN3W435AFD3HLSWQO4IADDFXPYL5V6GAOVYJTQ, seg_pos={0 0}, upload=2286]: Done uploading piece. err: ecclient: upload failed (node:1kovWN9SiUhcGQtZitjH9zgPj7iHaZwY5v37bjwPvArmgX44jA, address:104.36.82.54:4183): protocol: rpc: dial tcp 104.36.82.54:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:21:25.681761630 (2h29m41.462695007s) [noise=true , address=77.90.31.11:4574 , piece_id=WQUYLBLMJSEJBUN6OYGXH2WYJ3T2ANBTZEHNWMPHH2ZXCCG4SKTQ, seg_pos={0 0}, upload=2286]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: write tcp 159.223.143.39:36176->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:21:25.681806821 (2h29m41.462740201s) [noise=true , address=104.36.82.44:4874 , piece_id=YA7TS3PFJUES3YOS3S4PJTV6IO6TRTSNRUXNQ3JNA6Y22LDASQRA, seg_pos={0 0}, upload=2286]: Done uploading piece. err: ecclient: upload failed (node:12YApHMARaC7Wxeqopt72gpnyJS1xYBwhV7wbU9NJdR6W7hukyq, address:104.36.82.44:4874): protocol: rpc: dial tcp 104.36.82.44:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:21:25.681843169 (2h29m41.462776547s) [noise=true , address=104.36.82.47:4548 , piece_id=4XZC5P4LMAMYUGCXDV53VPPSJE2LHWHDPGVLB2YLPDWD2UR6JWVA, seg_pos={0 0}, upload=2286]: Done uploading piece. err: ecclient: upload failed (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT, address:104.36.82.47:4548): protocol: rpc: dial tcp 104.36.82.47:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:21:25.681915516 (2h29m41.462848894s) [noise=true , address=104.36.82.42:4816 , piece_id=ZRTNJ5HFG2O2TGDNJMISOCQBTZF6AWZ3JPTSDYQA5JCFVL7BKD7Q, seg_pos={0 0}, upload=2286]: Done uploading piece. err: ecclient: upload failed (node:1nq6pTeLCMEmtagczJkTMAdpADG5JrbSKhUjXAHy9oeS1pTpSx, address:104.36.82.42:4816): protocol: rpc: dial tcp 104.36.82.42:4816: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:21:25.681938875 (2h29m41.462872246s) [noise=true , address=104.36.82.52:4103 , piece_id=7XF3HHNNUEUM2XT7U72JN4ELTUSM4IJ7AQTWNKVOPBDAO7NG3VZA, seg_pos={0 0}, upload=2286]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: rpc: dial tcp 104.36.82.52:4103: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:21:25.682014743 (2h29m41.462948114s) [noise=true , address=104.36.82.15:4270 , piece_id=UXNWIMSPDEYXQCC6GVZRBNHFHLEYNIBCEFMUQFZ3IEEZIDFPPOFA, seg_pos={0 0}, upload=2286]: Done uploading piece. err: ecclient: upload failed (node:1nkuzy3vxeLdHhVs73Nceje7qXDiGJ1zEhpGgGKSk3cKb7vJbB, address:104.36.82.15:4270): protocol: write tcp 159.223.143.39:40398->104.36.82.15:4270: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:21:25.682067892 (2h29m41.463001279s) [seg_pos={0 0}, upload=2286]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Jul 27 00:21:25.709020105 (2h29m41.489953476s) [upload=2286]: Done uploading segments...
Jul 27 00:21:25.709070623 (2h29m41.490004005s) [upload=2286]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 1m0.02984712s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [X.XXX...............X..X......X.XX....X...X.................] duration:
2.589557223s
FAIL (2025-07-27 00:21:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.589589532s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
40 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.] duration:
2.443810874s
FAIL (2025-07-27 00:20:59 UTC)
orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 2.443859293s
core
uplink
us1
vivint
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
43 out of 60 is failed (threshold=3) [XXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....X] duration:
30.03066556s
FAIL (2025-07-27 00:20:57 UTC)
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
k connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:19:41.004485343 (2h27m56.785418722s) [seg_pos={0 0}, upload=2273]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jul 27 00:19:41.004577727 (2h27m56.785511106s) [upload=2273]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 27 00:19:41.004584582 (2h27m56.785517962s) [upload=2273]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 27 00:19:41.056659105 (2h27m56.83759247s) [upload=2273]: Done uploading segments...
Jul 27 00:19:41.056683745 (2h27m56.837617107s) [upload=2273]: Upload finished. err: <nil>
Jul 27 00:20:27.035648791 (2h28m42.816582178s) [upload=2287]: Starting upload
Jul 27 00:20:27.035802988 (2h28m42.816736349s) [upload=2287]: Done starting upload
Jul 27 00:20:27.036057893 (2h28m42.81699126s) [upload=2287]: Uploading segments...
Jul 27 00:20:27.036395184 (2h28m42.817328559s) [upload=2287]: Got next segment. Inline: false
Jul 27 00:20:27.036466520 (2h28m42.817399896s) [upload=2287]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 27 00:20:27.036475002 (2h28m42.817408382s) [upload=2287]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 27 00:20:27.066900323 (2h28m42.847833705s) [seg_pos={0 0}, upload=2287]: Begin upload segment...
Jul 27 00:20:27.066956201 (2h28m42.847889583s) [seg_pos={0 0}, upload=2287]: Done begin upload segment.
Jul 27 00:20:27.066990267 (2h28m42.84792366s) [noise=true , address=45.91.116.7:4009 , piece_id=C6UBSKL6EUJKUHGBV3B3CV5KOVARUAOOVZWXK3HUQDPDEW3UF6PQ, seg_pos={0 0}, upload=2287]: Uploading piece...
Jul 27 00:20:27.067074462 (2h28m42.848007846s) [noise=true , address=77.90.31.18:4807 , piece_id=PPTKCS3RZFAF5O7LB66WYY7KGREPDZ5KU55QZHUF4M72DYCDNTQQ, seg_pos={0 0}, upload=2287]: Uploading piece...
Jul 27 00:20:27.067470705 (2h28m42.848404098s) [noise=true , address=45.91.116.6:4975 , piece_id=BEDHQWJQ5HVFDVI3GYQL2T33YGNPFCLRZYKXZHXRY56O4CIN277Q, seg_pos={0 0}, upload=2287]: Uploading piece...
Jul 27 00:20:27.067643399 (2h28m42.848576785s) [noise=true , address=77.90.31.11:4553 , piece_id=4NQHV2AWCLHCH2H7OZ4OWTGIHZFGTGLEUAKULOE3VB7TJFJ3DXHQ, seg_pos={0 0}, upload=2287]: Uploading piece...
Jul 27 00:20:27.067915059 (2h28m42.848848443s) [noise=true , address=45.91.116.11:4119 , piece_id=GFXD2IHI6EIM4ZFNEWTEQKRXCQJT5UDPOJ4DNZT7P6JYT54354EA, seg_pos={0 0}, upload=2287]: Uploading piece...
Jul 27 00:20:27.068285088 (2h28m42.849218476s) [noise=true , address=45.91.116.8:4045 , piece_id=K4KJEGN32V674LIOFRBDOVHC2MBMDE7MG6EXDULE2SKNHGCSPTCQ, seg_pos={0 0}, upload=2287]: Uploading piece...
Jul 27 00:20:27.068446646 (2h28m42.849380041s) [noise=true , address=45.91.116.4:4939 , piece_id=IDRX5BRF4KGKWHZLESPCCTNV6VNPFMN7AFOKLCN7HSNZGISJFONA, seg_pos={0 0}, upload=2287]: Uploading piece...
Jul 27 00:20:27.068773204 (2h28m42.849706584s) [noise=true , address=77.90.31.14:4663 , piece_id=KUVPM5352PFA6CNYHZQWPKRPBMSG5Q5JHAZUKVR7TE3JIOAOL4OA, seg_pos={0 0}, upload=2287]: Uploading piece...
Jul 27 00:20:27.068834037 (2h28m42.849767424s) [noise=true , address=45.91.116.12:4225 , piece_id=FZKCX7WR2Y2H2V2AFGHV2JMUPAI2PN6HV2PADQKOYQR7KRHAIE5Q, seg_pos={0 0}, upload=2287]: Uploading piece...
Jul 27 00:20:27.069101011 (2h28m42.850034399s) [noise=true , address=77.90.31.21:4877 , piece_id=67JFI32F7BX2ZNADJPNMT4XWE3P6IRNYNWEKZV3VUF5KQRAPP2MQ, seg_pos={0 0}, upload=2287]: Uploading piece...
Jul 27 00:20:27.069390923 (2h28m42.850324317s) [noise=true , address=104.36.82.4:4802 , piece_id=7IJO6A2ZCUFYMMTTJJXLZDXKXMSTKDDQA3M6G6OHTF7BMN65PUTQ, seg_pos={0 0}, upload=2287]: Uploading piece...
Jul 27 00:20:27.069417578 (2h28m42.850350958s) [noise=true , address=104.36.82.9:4006 , piece_id=YTL6SHGOFZ4QGPVKLAIZQOD26S23LK3DQTPFBYP54WAO7QSMCAAQ, seg_pos={0 0}, upload=2287]: Uploading piece...
Jul 27 00:20:27.069706097 (2h28m42.850639476s) [noise=true , address=104.36.82.16:4294 , piece_id=CJOUPJLI7ICUZPCOPTPCYTOZVBAIHEZOYH3O33PCWTGS2UIRLZ7A, seg_pos={0 0}, upload=2287]: Uploading piece...
Jul 27 00:20:27.069955506 (2h28m42.850888893s) [noise=true , address=104.36.82.16:4293 , piece_id=APXIGGCECRH2KEM5B7OI7KDWNP43K5VGGLQEPEUTZQYCYZSTCHSA, seg_pos={0 0}, upload=2287]: Uploading piece...
Jul 27 00:20:27.070498452 (2h28m42.851431831s) [noise=true , address=45.91.116.9:4263 , piece_id=56D2LMCWEMUQB7BW5AV75ZB4EAIWFWUXW2ASMPA6V64UEIVI6OVQ, seg_pos={0 0}, upload=2287]: Uploading piece...
Jul 27 00:20:27.071135645 (2h28m42.85206909s) [noise=true , address=45.91.116.13:4155 , piece_id=QQALYGTDCSJPM7ZPS3BKWIQHBQPEI5XGJOV4ULFDMWGJ7QSF4T6Q, seg_pos={0 0}, upload=2287]: Uploading piece...
Jul 27 00:20:27.170875724 (2h28m42.951809111s) [upload=2287]: Next returned nil segment
Jul 27 00:20:27.170898320 (2h28m42.95183171s) [upload=2287]: Waiting for error group managing segments...
Jul 27 00:20:27.174860754 (2h28m42.955794136s) [noise=true , address=45.91.116.11:4119 , piece_id=GFXD2IHI6EIM4ZFNEWTEQKRXCQJT5UDPOJ4DNZT7P6JYT54354EA, seg_pos={0 0}, upload=2287]: Done uploading piece. err: <nil>
Jul 27 00:20:27.175748787 (2h28m42.956682163s) [noise=true , address=45.91.116.12:4225 , piece_id=FZKCX7WR2Y2H2V2AFGHV2JMUPAI2PN6HV2PADQKOYQR7KRHAIE5Q, seg_pos={0 0}, upload=2287]: Done uploading piece. err: <nil>
Jul 27 00:20:27.176640849 (2h28m42.957574228s) [noise=true , address=45.91.116.8:4045 , piece_id=K4KJEGN32V674LIOFRBDOVHC2MBMDE7MG6EXDULE2SKNHGCSPTCQ, seg_pos={0 0}, upload=2287]: Done uploading piece. err: <nil>
Jul 27 00:20:27.176806409 (2h28m42.957739778s) [noise=true , address=77.90.31.18:4807 , piece_id=PPTKCS3RZFAF5O7LB66WYY7KGREPDZ5KU55QZHUF4M72DYCDNTQQ, seg_pos={0 0}, upload=2287]: Done uploading piece. err: <nil>
Jul 27 00:20:27.177168999 (2h28m42.95810238s) [noise=true , address=77.90.31.14:4663 , piece_id=KUVPM5352PFA6CNYHZQWPKRPBMSG5Q5JHAZUKVR7TE3JIOAOL4OA, seg_pos={0 0}, upload=2287]: Done uploading piece. err: <nil>
Jul 27 00:20:27.177386831 (2h28m42.958320201s) [noise=true , address=45.91.116.9:4263 , piece_id=56D2LMCWEMUQB7BW5AV75ZB4EAIWFWUXW2ASMPA6V64UEIVI6OVQ, seg_pos={0 0}, upload=2287]: Done uploading piece. err: <nil>
Jul 27 00:20:27.177693567 (2h28m42.958626953s) [noise=true , address=45.91.116.4:4939 , piece_id=IDRX5BRF4KGKWHZLESPCCTNV6VNPFMN7AFOKLCN7HSNZGISJFONA, seg_pos={0 0}, upload=2287]: Done uploading piece. err: <nil>
Jul 27 00:20:27.177951420 (2h28m42.958884796s) [noise=true , address=45.91.116.6:4975 , piece_id=BEDHQWJQ5HVFDVI3GYQL2T33YGNPFCLRZYKXZHXRY56O4CIN277Q, seg_pos={0 0}, upload=2287]: Done uploading piece. err: <nil>
Jul 27 00:20:27.178240212 (2h28m42.959173595s) [noise=true , address=45.91.116.7:4009 , piece_id=C6UBSKL6EUJKUHGBV3B3CV5KOVARUAOOVZWXK3HUQDPDEW3UF6PQ, seg_pos={0 0}, upload=2287]: Done uploading piece. err: <nil>
Jul 27 00:20:27.178506692 (2h28m42.959440062s) [noise=true , address=45.91.116.13:4155 , piece_id=QQALYGTDCSJPM7ZPS3BKWIQHBQPEI5XGJOV4ULFDMWGJ7QSF4T6Q, seg_pos={0 0}, upload=2287]: Done uploading piece. err: <nil>
Jul 27 00:20:57.006233759 (2h29m12.78716714s) [noise=true , address=104.36.82.9:4006 , piece_id=YTL6SHGOFZ4QGPVKLAIZQOD26S23LK3DQTPFBYP54WAO7QSMCAAQ, seg_pos={0 0}, upload=2287]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:20:57.006303901 (2h29m12.787237291s) [noise=true , address=104.36.82.16:4293 , piece_id=APXIGGCECRH2KEM5B7OI7KDWNP43K5VGGLQEPEUTZQYCYZSTCHSA, seg_pos={0 0}, upload=2287]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:20:57.006352195 (2h29m12.78728557s) [noise=true , address=104.36.82.4:4802 , piece_id=7IJO6A2ZCUFYMMTTJJXLZDXKXMSTKDDQA3M6G6OHTF7BMN65PUTQ, seg_pos={0 0}, upload=2287]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 159.223.143.39:39186->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:20:57.006369374 (2h29m12.787302744s) [noise=true , address=104.36.82.16:4294 , piece_id=CJOUPJLI7ICUZPCOPTPCYTOZVBAIHEZOYH3O33PCWTGS2UIRLZ7A, seg_pos={0 0}, upload=2287]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:20:57.006407241 (2h29m12.787340611s) [noise=true , address=77.90.31.11:4553 , piece_id=4NQHV2AWCLHCH2H7OZ4OWTGIHZFGTGLEUAKULOE3VB7TJFJ3DXHQ, seg_pos={0 0}, upload=2287]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: rpc: dial tcp 77.90.31.11:4553: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 27 00:20:57.006580930 (2h29m12.787514304s) [noise=true , address=77.90.31.21:4877 , piece_id=67JFI32F7BX2ZNADJPNMT4XWE3P6IRNYNWEKZV3VUF5KQRAPP2MQ, seg_pos={0 0}, upload=2287]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded
Jul 27 00:20:57.006650182 (2h29m12.787583572s) [seg_pos={0 0}, upload=2287]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 27 00:20:57.034251290 (2h29m12.815184674s) [upload=2287]: Done uploading segments...
Jul 27 00:20:57.034293667 (2h29m12.815227031s) [upload=2287]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 30.030800625s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [.X.XXX...............X..X......X.XX....X...X................] duration:
21.682671438s
FAIL (2025-07-27 00:20:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 21.682706562s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.............X.X........X...XX..............................] duration:
17.728662562s
FAIL (2025-07-27 00:20:45 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 17.72868333s
core
uplink
us-select-1
us1-placement-20
|