Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [....X.XXX.XX...............................................X] duration: 1m0.001286519s FAIL (2025-08-07 23:54:22 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001306819s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [......XXX..X.X.X.....................................X......] duration: 529.041961ms FAIL (2025-08-07 23:54:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 529.652918ms edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 46.821468ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 23:54:20 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 46.843154ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 10 out of 60 is failed (threshold=3) [.X..XXXXX.X.XX...........................................X..] duration: 288.52869ms FAIL (2025-08-07 23:54:14 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 288.554458ms core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [...XX.X.X....XX..........................................X..] duration: 360.932931ms FAIL (2025-08-07 23:54:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 360.954614ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 5 out of 60 is failed (threshold=3) [....XX..X...X..............................................X] duration: 1m0.00140631s FAIL (2025-08-07 23:54:08 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .485423327s) [upload=14510]: Done uploading segments... Aug 7 23:45:37.970340911 (14h6m12.48544349s) [upload=14510]: Upload finished. err: <nil> Aug 7 23:46:02.549574682 (14h6m37.064677251s) [upload=14516]: Starting upload Aug 7 23:46:02.549657903 (14h6m37.064760484s) [upload=14516]: Done starting upload Aug 7 23:46:02.549720180 (14h6m37.064822746s) [upload=14516]: Uploading segments... Aug 7 23:46:02.549756468 (14h6m37.064859054s) [upload=14516]: Got next segment. Inline: true Aug 7 23:46:02.549761926 (14h6m37.064864504s) [upload=14516]: Waiting for error group managing segments... Aug 7 23:46:02.549820984 (14h6m37.064923564s) [upload=14516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:46:02.549827027 (14h6m37.064929607s) [upload=14516]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:46:02.549831859 (14h6m37.06493444s) [upload=14516]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:46:02.612412597 (14h6m37.127515192s) [upload=14516]: Done uploading segments... Aug 7 23:46:02.612451273 (14h6m37.127553857s) [upload=14516]: Upload finished. err: <nil> Aug 7 23:46:33.174008493 (14h7m7.68911107s) [upload=14523]: Starting upload Aug 7 23:46:33.174095566 (14h7m7.689198146s) [upload=14523]: Done starting upload Aug 7 23:46:33.174169016 (14h7m7.689271584s) [upload=14523]: Uploading segments... Aug 7 23:46:33.174213635 (14h7m7.68931622s) [upload=14523]: Got next segment. Inline: true Aug 7 23:46:33.174220067 (14h7m7.689322652s) [upload=14523]: Waiting for error group managing segments... Aug 7 23:46:33.174271363 (14h7m7.689373942s) [upload=14523]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:46:33.174277423 (14h7m7.689380007s) [upload=14523]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:46:33.174281768 (14h7m7.689384349s) [upload=14523]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:46:33.410961160 (14h7m7.926063757s) [upload=14523]: Done uploading segments... Aug 7 23:46:33.410982034 (14h7m7.926084613s) [upload=14523]: Upload finished. err: <nil> Aug 7 23:47:43.879457284 (14h8m18.394559871s) [upload=14540]: Starting upload Aug 7 23:47:43.879610635 (14h8m18.394713221s) [upload=14540]: Done starting upload Aug 7 23:47:43.879713914 (14h8m18.394816491s) [upload=14540]: Uploading segments... Aug 7 23:47:43.879808496 (14h8m18.39491109s) [upload=14540]: Got next segment. Inline: true Aug 7 23:47:43.879816280 (14h8m18.394918867s) [upload=14540]: Waiting for error group managing segments... Aug 7 23:47:43.879892463 (14h8m18.394995049s) [upload=14540]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:47:43.879898862 (14h8m18.395001442s) [upload=14540]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:47:43.879902786 (14h8m18.395005367s) [upload=14540]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:47:43.948673039 (14h8m18.463775606s) [upload=14540]: Done uploading segments... Aug 7 23:47:43.948690272 (14h8m18.463792838s) [upload=14540]: Upload finished. err: <nil> Aug 7 23:48:32.733662762 (14h9m7.248765339s) [upload=14555]: Starting upload Aug 7 23:48:32.733764614 (14h9m7.248867195s) [upload=14555]: Done starting upload Aug 7 23:48:32.733839718 (14h9m7.248942295s) [upload=14555]: Uploading segments... Aug 7 23:48:32.733888071 (14h9m7.248990659s) [upload=14555]: Got next segment. Inline: true Aug 7 23:48:32.733893767 (14h9m7.248996349s) [upload=14555]: Waiting for error group managing segments... Aug 7 23:48:32.733943638 (14h9m7.249046215s) [upload=14555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:48:32.733949815 (14h9m7.249052399s) [upload=14555]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:48:32.733954110 (14h9m7.24905669s) [upload=14555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:48:32.809665685 (14h9m7.324768252s) [upload=14555]: Done uploading segments... Aug 7 23:48:32.809699832 (14h9m7.324802408s) [upload=14555]: Upload finished. err: <nil> Aug 7 23:49:43.381878590 (14h10m17.896981166s) [upload=14573]: Starting upload Aug 7 23:49:43.382020923 (14h10m17.897123495s) [upload=14573]: Done starting upload Aug 7 23:49:43.382128913 (14h10m17.897231484s) [upload=14573]: Uploading segments... Aug 7 23:49:43.382204063 (14h10m17.897306641s) [upload=14573]: Got next segment. Inline: true Aug 7 23:49:43.382211147 (14h10m17.897313718s) [upload=14573]: Waiting for error group managing segments... Aug 7 23:49:43.382287220 (14h10m17.897389786s) [upload=14573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:49:43.382292849 (14h10m17.897395421s) [upload=14573]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:49:43.382297255 (14h10m17.897399833s) [upload=14573]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:49:43.532367962 (14h10m18.047470542s) [upload=14573]: Done uploading segments... Aug 7 23:49:43.532399853 (14h10m18.047502429s) [upload=14573]: Upload finished. err: <nil> Aug 7 23:50:25.794985700 (14h11m0.310088267s) [upload=14588]: Starting upload Aug 7 23:50:25.795091135 (14h11m0.310193704s) [upload=14588]: Done starting upload Aug 7 23:50:25.795192505 (14h11m0.310295073s) [upload=14588]: Uploading segments... Aug 7 23:50:25.795250709 (14h11m0.310353286s) [upload=14588]: Got next segment. Inline: true Aug 7 23:50:25.795256599 (14h11m0.310359177s) [upload=14588]: Waiting for error group managing segments... Aug 7 23:50:25.795298950 (14h11m0.310401527s) [upload=14588]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:25.795303481 (14h11m0.310406058s) [upload=14588]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:25.795306901 (14h11m0.310409476s) [upload=14588]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:25.886991014 (14h11m0.402093596s) [upload=14588]: Done uploading segments... Aug 7 23:50:25.887011651 (14h11m0.402114228s) [upload=14588]: Upload finished. err: <nil> Aug 7 23:51:02.166684989 (14h11m36.681787571s) [upload=14598]: Starting upload Aug 7 23:51:02.166793315 (14h11m36.681895904s) [upload=14598]: Done starting upload Aug 7 23:51:02.167617113 (14h11m36.682719692s) [upload=14598]: Uploading segments... Aug 7 23:51:02.167688699 (14h11m36.682791288s) [upload=14598]: Got next segment. Inline: true Aug 7 23:51:02.167696965 (14h11m36.68279955s) [upload=14598]: Waiting for error group managing segments... Aug 7 23:51:02.167768986 (14h11m36.682871569s) [upload=14598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:02.167776149 (14h11m36.682878967s) [upload=14598]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:02.167781308 (14h11m36.682883888s) [upload=14598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:02.249738740 (14h11m36.764841311s) [upload=14598]: Done uploading segments... Aug 7 23:51:02.249754857 (14h11m36.764857435s) [upload=14598]: Upload finished. err: <nil> Aug 7 23:51:48.536453050 (14h12m23.051555623s) [upload=14610]: Starting upload Aug 7 23:51:48.536534615 (14h12m23.051637193s) [upload=14610]: Done starting upload Aug 7 23:51:48.536600634 (14h12m23.051703211s) [upload=14610]: Uploading segments... Aug 7 23:51:48.536652707 (14h12m23.051755293s) [upload=14610]: Got next segment. Inline: true Aug 7 23:51:48.536658284 (14h12m23.051760862s) [upload=14610]: Waiting for error group managing segments... Aug 7 23:51:48.536705902 (14h12m23.051808486s) [upload=14610]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:48.536710850 (14h12m23.051813429s) [upload=14610]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:51:48.536714377 (14h12m23.051816954s) [upload=14610]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:48.601979182 (14h12m23.117081769s) [upload=14610]: Done uploading segments... Aug 7 23:51:48.602001648 (14h12m23.117104229s) [upload=14610]: Upload finished. err: <nil> Aug 7 23:52:04.519337006 (14h12m39.034439591s) [upload=14619]: Starting upload Aug 7 23:52:04.519460694 (14h12m39.034563258s) [upload=14619]: Done starting upload Aug 7 23:52:04.519569621 (14h12m39.034672186s) [upload=14619]: Uploading segments... Aug 7 23:52:04.519675504 (14h12m39.034778091s) [upload=14619]: Got next segment. Inline: true Aug 7 23:52:04.519701072 (14h12m39.034803649s) [upload=14619]: Waiting for error group managing segments... Aug 7 23:52:04.519755555 (14h12m39.034858131s) [upload=14619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:04.519760650 (14h12m39.034863226s) [upload=14619]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:04.519777078 (14h12m39.034879654s) [upload=14619]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:04.588422656 (14h12m39.103525241s) [upload=14619]: Done uploading segments... Aug 7 23:52:04.588444719 (14h12m39.103547297s) [upload=14619]: Upload finished. err: <nil> Aug 7 23:53:08.147087231 (14h13m42.662189804s) [upload=14641]: Starting upload Aug 7 23:53:08.147181411 (14h13m42.662283996s) [upload=14641]: Done starting upload Aug 7 23:53:08.147281202 (14h13m42.66238377s) [upload=14641]: Uploading segments... Aug 7 23:53:08.147322830 (14h13m42.662425415s) [upload=14641]: Got next segment. Inline: true Aug 7 23:53:08.147327701 (14h13m42.662430277s) [upload=14641]: Waiting for error group managing segments... Aug 7 23:53:08.147455761 (14h13m42.66255833s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:08.147460709 (14h13m42.662563285s) [upload=14641]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:08.147463890 (14h13m42.662566457s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:08.116969456 (14h14m42.632072039s) [upload=14641]: Done uploading segments... Aug 7 23:54:08.117007036 (14h14m42.632109618s) [upload=14641]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001428984s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 10 out of 60 is failed (threshold=3) [..X..XXXXX.X.XX...........................................X.] duration: 262.132771ms FAIL (2025-08-07 23:53:56 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 262.163228ms core uplink us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 30.335163ms target: https://us1.storj.io/login FAIL (2025-08-07 23:53:53 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 30.359414ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 10 out of 60 is failed (threshold=3) [...X..XXXXX.X.XX...........................................X] duration: 30.003057888s FAIL (2025-08-07 23:53:47 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.003091895s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [....XX.X.X....XX..........................................X.] duration: 250.189305ms FAIL (2025-08-07 23:53:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 250.211126ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [.......XX.XXXX...X.X......................................X.] duration: 1.142309138s FAIL (2025-08-07 23:53:46 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== em_SegmentBegin Aug 7 23:52:53.694597728 (14h13m28.209700293s) [seg_pos={0 0}, upload=14631]: Begin upload segment... Aug 7 23:52:53.694678279 (14h13m28.209780847s) [seg_pos={0 0}, upload=14631]: Done begin upload segment. Aug 7 23:52:53.695313906 (14h13m28.210416492s) [noise=true , address=104.36.82.16:4296 , piece_id=SPEE6NAFLQF74MDZYA7K6IBVUILHAWOKJSBSGKF6QJTBJ6CGDGBA, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.695376820 (14h13m28.210479485s) [noise=true , address=104.36.82.43:4838 , piece_id=KLNUL263DQQUZ5BJQTBFUZEVYXKVUFMSLHERH6LBNDIXMJWZ23KQ, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.696012096 (14h13m28.21111467s) [noise=true , address=104.36.82.35:4590 , piece_id=4USMUKHDNE2WBPE5GGEB3P4SC77CCFLLETXVPPOXJ2ZUNTDER7ZQ, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.696078398 (14h13m28.211180979s) [noise=true , address=104.36.82.44:4898 , piece_id=3PB72NHDN3PIPM4VMYPOELLIMMYOMCM3KOE2A77F4VVYIWQWMNGA, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.696381146 (14h13m28.211483715s) [noise=true , address=104.36.82.45:4926 , piece_id=PGKQGEMNS2AP423PUETAZZIXNNI7ZTQOWY6SGXDWVQ3DMRQECQGQ, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.696482041 (14h13m28.211584622s) [noise=true , address=79.127.201.215:4356 , piece_id=OSQLCO2P5NKWKD5KGEX2KW6UNC3XUE6KYBDWH47NH52IHG7IXOUQ, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.696707399 (14h13m28.211809981s) [noise=true , address=104.36.82.23:4459 , piece_id=PP6FJQDJ3TMKUYF4LEDPR4JRA5GZ6ZDMC24M6BVTGDH4DFSS6GIQ, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.696874176 (14h13m28.211976755s) [noise=true , address=104.36.82.9:4028 , piece_id=X63TSF2TVUZXVEAL2RA3IMSXF7VCXHA442QJCCJEMNTC2YIWZMWA, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.697034811 (14h13m28.212137396s) [noise=true , address=104.36.82.52:4098 , piece_id=RXWXDTPQUUCJ57XTCO3TQOBSYDFQ4MZGH7O6E3OFMZULLN3P7ITQ, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.697282512 (14h13m28.212385095s) [noise=true , address=45.91.116.4:4964 , piece_id=AAEEGFTLA3USENXJPUS2UD6SW2YTP4GMOAOO4GCBH2AYYT73DFAQ, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.697380449 (14h13m28.212483028s) [noise=true , address=104.36.82.10:4109 , piece_id=24NFW3W3KBWD5GPQXNTICBF5YHFVLQLZ7IMJOZNR4DKNTONSY4HA, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.697699829 (14h13m28.212802405s) [noise=true , address=104.36.82.4:4824 , piece_id=F4YOXTKWOGGT5QHYXFIZL4FJOOVHBKEGHHEJXI2ZX2HJ6ZKYN5CQ, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.698042345 (14h13m28.213144928s) [noise=true , address=104.36.82.54:4165 , piece_id=KQ776X5RH6VYMMEZ2WGEAHFIQVF6257XLIOOQL7ZPPDBGNUD4PYA, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.698350247 (14h13m28.213452827s) [noise=true , address=104.36.82.39:4746 , piece_id=NMKV5KBU3V2KVSFRITKGNFPQISKIFK3RDKJUZQSH3MBU26MXHDRA, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.699601440 (14h13m28.214704028s) [noise=true , address=104.36.82.47:4525 , piece_id=BE22FSLCHMRFEWKAWWBP3I7KUO5PFQGWSJEVTBTFT223GCRWOW2Q, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.700693686 (14h13m28.215796268s) [noise=true , address=104.36.82.46:4515 , piece_id=AMUEDZI5RDYCSKAT6LDWLPBSJ2GCPFM2FCWTESJISYPN3IBC3SCA, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.701810303 (14h13m28.21691288s) [noise=true , address=104.36.82.39:4741 , piece_id=JWBNNZNT3ZOTCDKCV7EG4GECH422LP5CDRHL4CFGTJ2EMTNLJN5A, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.702831457 (14h13m28.217934039s) [noise=true , address=45.91.116.9:4290 , piece_id=TFE3T2MFGXLT4GLD6JBTE73MHIC7QGGNIUWDS7LKM2D4CMQZIOYA, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.767754625 (14h13m28.282857196s) [upload=14631]: Next returned nil segment Aug 7 23:52:53.767768780 (14h13m28.282871345s) [upload=14631]: Waiting for error group managing segments... Aug 7 23:52:53.777227926 (14h13m28.292330496s) [noise=true , address=104.36.82.39:4746 , piece_id=NMKV5KBU3V2KVSFRITKGNFPQISKIFK3RDKJUZQSH3MBU26MXHDRA, seg_pos={0 0}, upload=14631]: Done uploading piece. err: <nil> Aug 7 23:52:53.778525320 (14h13m28.293627897s) [noise=true , address=104.36.82.46:4515 , piece_id=AMUEDZI5RDYCSKAT6LDWLPBSJ2GCPFM2FCWTESJISYPN3IBC3SCA, seg_pos={0 0}, upload=14631]: Done uploading piece. err: <nil> Aug 7 23:52:53.788240783 (14h13m28.303343347s) [noise=true , address=104.36.82.16:4296 , piece_id=SPEE6NAFLQF74MDZYA7K6IBVUILHAWOKJSBSGKF6QJTBJ6CGDGBA, seg_pos={0 0}, upload=14631]: Done uploading piece. err: <nil> Aug 7 23:52:53.789097991 (14h13m28.304200562s) [noise=true , address=104.36.82.39:4741 , piece_id=JWBNNZNT3ZOTCDKCV7EG4GECH422LP5CDRHL4CFGTJ2EMTNLJN5A, seg_pos={0 0}, upload=14631]: Done uploading piece. err: <nil> Aug 7 23:52:53.790182855 (14h13m28.305285426s) [noise=true , address=45.91.116.9:4290 , piece_id=TFE3T2MFGXLT4GLD6JBTE73MHIC7QGGNIUWDS7LKM2D4CMQZIOYA, seg_pos={0 0}, upload=14631]: Done uploading piece. err: <nil> Aug 7 23:52:53.791086415 (14h13m28.306188981s) [noise=true , address=45.91.116.4:4964 , piece_id=AAEEGFTLA3USENXJPUS2UD6SW2YTP4GMOAOO4GCBH2AYYT73DFAQ, seg_pos={0 0}, upload=14631]: Done uploading piece. err: <nil> Aug 7 23:52:53.792514550 (14h13m28.307617117s) [noise=true , address=104.36.82.54:4165 , piece_id=KQ776X5RH6VYMMEZ2WGEAHFIQVF6257XLIOOQL7ZPPDBGNUD4PYA, seg_pos={0 0}, upload=14631]: Done uploading piece. err: <nil> Aug 7 23:52:53.806891024 (14h13m28.321993602s) [noise=true , address=104.36.82.52:4098 , piece_id=RXWXDTPQUUCJ57XTCO3TQOBSYDFQ4MZGH7O6E3OFMZULLN3P7ITQ, seg_pos={0 0}, upload=14631]: Done uploading piece. err: <nil> Aug 7 23:52:53.811849722 (14h13m28.326952306s) [noise=true , address=104.36.82.4:4824 , piece_id=F4YOXTKWOGGT5QHYXFIZL4FJOOVHBKEGHHEJXI2ZX2HJ6ZKYN5CQ, seg_pos={0 0}, upload=14631]: Done uploading piece. err: <nil> Aug 7 23:52:53.827462777 (14h13m28.342565345s) [noise=true , address=104.36.82.23:4459 , piece_id=PP6FJQDJ3TMKUYF4LEDPR4JRA5GZ6ZDMC24M6BVTGDH4DFSS6GIQ, seg_pos={0 0}, upload=14631]: Done uploading piece. err: <nil> Aug 7 23:52:53.831869285 (14h13m28.346971865s) [noise=true , address=104.36.82.47:4525 , piece_id=BE22FSLCHMRFEWKAWWBP3I7KUO5PFQGWSJEVTBTFT223GCRWOW2Q, seg_pos={0 0}, upload=14631]: Done uploading piece. err: <nil> Aug 7 23:52:53.837103172 (14h13m28.352205754s) [noise=true , address=104.36.82.9:4028 , piece_id=X63TSF2TVUZXVEAL2RA3IMSXF7VCXHA442QJCCJEMNTC2YIWZMWA, seg_pos={0 0}, upload=14631]: Done uploading piece. err: <nil> Aug 7 23:52:53.982157141 (14h13m28.497259719s) [noise=true , address=104.36.82.44:4898 , piece_id=3PB72NHDN3PIPM4VMYPOELLIMMYOMCM3KOE2A77F4VVYIWQWMNGA, seg_pos={0 0}, upload=14631]: Done uploading piece. err: <nil> Aug 7 23:52:54.036537934 (14h13m28.551640504s) [noise=true , address=79.127.201.215:4356 , piece_id=OSQLCO2P5NKWKD5KGEX2KW6UNC3XUE6KYBDWH47NH52IHG7IXOUQ, seg_pos={0 0}, upload=14631]: Done uploading piece. err: <nil> Aug 7 23:52:54.036561698 (14h13m28.551664274s) [seg_pos={0 0}, upload=14631]: Segment reached optimal threshold of 14 pieces. Aug 7 23:52:54.036864257 (14h13m28.551966838s) [noise=true , address=104.36.82.43:4838 , piece_id=KLNUL263DQQUZ5BJQTBFUZEVYXKVUFMSLHERH6LBNDIXMJWZ23KQ, seg_pos={0 0}, upload=14631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eorQLNmgEaXv8myr6t3Q8rysGwsvtdZzu5b7sndhoANb6Waj6): protocol: expected piece hash; context canceled; EOF Aug 7 23:52:54.037055935 (14h13m28.552158502s) [noise=true , address=104.36.82.10:4109 , piece_id=24NFW3W3KBWD5GPQXNTICBF5YHFVLQLZ7IMJOZNR4DKNTONSY4HA, seg_pos={0 0}, upload=14631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZE1ut9Tum4tzq2m8zbFaGUNLdNZSZQYARkayDAfE1b7uZtWjW): protocol: write tcp 159.223.143.39:48750->104.36.82.10:4109: use of closed network connection; write tcp 159.223.143.39:48750->104.36.82.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48750->104.36.82.10:4109: use of closed network connection Aug 7 23:52:54.037258369 (14h13m28.552360935s) [noise=true , address=104.36.82.35:4590 , piece_id=4USMUKHDNE2WBPE5GGEB3P4SC77CCFLLETXVPPOXJ2ZUNTDER7ZQ, seg_pos={0 0}, upload=14631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128t6B6PhCQznqwMTGJdkWuuCyqQ4bVHbPbZWWnYwb8Eyc1DjDw): protocol: write tcp 159.223.143.39:49450->104.36.82.35:4590: use of closed network connection; write tcp 159.223.143.39:49450->104.36.82.35:4590: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49450->104.36.82.35:4590: use of closed network connection Aug 7 23:52:54.037340084 (14h13m28.552442668s) [noise=true , address=104.36.82.45:4926 , piece_id=PGKQGEMNS2AP423PUETAZZIXNNI7ZTQOWY6SGXDWVQ3DMRQECQGQ, seg_pos={0 0}, upload=14631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LkGN9UdiF163eg78WUPpXqz4FxHBxhnPin4okcdU1YPDWx4MF): protocol: read tcp 159.223.143.39:32804->104.36.82.45:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:52:54.037370062 (14h13m28.552472628s) [seg_pos={0 0}, upload=14631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:52:54.037456923 (14h13m28.552559488s) [upload=14631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:52:54.037461311 (14h13m28.552563877s) [upload=14631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:13.558610345 (14h13m48.07371291s) [upload=14631]: Done uploading segments... Aug 7 23:53:13.558627739 (14h13m48.073730309s) [upload=14631]: Upload finished. err: <nil> Duration: 1.142331549s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [........XX.XXXX...X.X......................................X] duration: 1m0.002246319s FAIL (2025-08-07 23:53:44 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== em_SegmentBegin Aug 7 23:52:53.694597728 (14h13m28.209700293s) [seg_pos={0 0}, upload=14631]: Begin upload segment... Aug 7 23:52:53.694678279 (14h13m28.209780847s) [seg_pos={0 0}, upload=14631]: Done begin upload segment. Aug 7 23:52:53.695313906 (14h13m28.210416492s) [noise=true , address=104.36.82.16:4296 , piece_id=SPEE6NAFLQF74MDZYA7K6IBVUILHAWOKJSBSGKF6QJTBJ6CGDGBA, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.695376820 (14h13m28.210479485s) [noise=true , address=104.36.82.43:4838 , piece_id=KLNUL263DQQUZ5BJQTBFUZEVYXKVUFMSLHERH6LBNDIXMJWZ23KQ, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.696012096 (14h13m28.21111467s) [noise=true , address=104.36.82.35:4590 , piece_id=4USMUKHDNE2WBPE5GGEB3P4SC77CCFLLETXVPPOXJ2ZUNTDER7ZQ, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.696078398 (14h13m28.211180979s) [noise=true , address=104.36.82.44:4898 , piece_id=3PB72NHDN3PIPM4VMYPOELLIMMYOMCM3KOE2A77F4VVYIWQWMNGA, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.696381146 (14h13m28.211483715s) [noise=true , address=104.36.82.45:4926 , piece_id=PGKQGEMNS2AP423PUETAZZIXNNI7ZTQOWY6SGXDWVQ3DMRQECQGQ, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.696482041 (14h13m28.211584622s) [noise=true , address=79.127.201.215:4356 , piece_id=OSQLCO2P5NKWKD5KGEX2KW6UNC3XUE6KYBDWH47NH52IHG7IXOUQ, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.696707399 (14h13m28.211809981s) [noise=true , address=104.36.82.23:4459 , piece_id=PP6FJQDJ3TMKUYF4LEDPR4JRA5GZ6ZDMC24M6BVTGDH4DFSS6GIQ, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.696874176 (14h13m28.211976755s) [noise=true , address=104.36.82.9:4028 , piece_id=X63TSF2TVUZXVEAL2RA3IMSXF7VCXHA442QJCCJEMNTC2YIWZMWA, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.697034811 (14h13m28.212137396s) [noise=true , address=104.36.82.52:4098 , piece_id=RXWXDTPQUUCJ57XTCO3TQOBSYDFQ4MZGH7O6E3OFMZULLN3P7ITQ, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.697282512 (14h13m28.212385095s) [noise=true , address=45.91.116.4:4964 , piece_id=AAEEGFTLA3USENXJPUS2UD6SW2YTP4GMOAOO4GCBH2AYYT73DFAQ, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.697380449 (14h13m28.212483028s) [noise=true , address=104.36.82.10:4109 , piece_id=24NFW3W3KBWD5GPQXNTICBF5YHFVLQLZ7IMJOZNR4DKNTONSY4HA, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.697699829 (14h13m28.212802405s) [noise=true , address=104.36.82.4:4824 , piece_id=F4YOXTKWOGGT5QHYXFIZL4FJOOVHBKEGHHEJXI2ZX2HJ6ZKYN5CQ, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.698042345 (14h13m28.213144928s) [noise=true , address=104.36.82.54:4165 , piece_id=KQ776X5RH6VYMMEZ2WGEAHFIQVF6257XLIOOQL7ZPPDBGNUD4PYA, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.698350247 (14h13m28.213452827s) [noise=true , address=104.36.82.39:4746 , piece_id=NMKV5KBU3V2KVSFRITKGNFPQISKIFK3RDKJUZQSH3MBU26MXHDRA, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.699601440 (14h13m28.214704028s) [noise=true , address=104.36.82.47:4525 , piece_id=BE22FSLCHMRFEWKAWWBP3I7KUO5PFQGWSJEVTBTFT223GCRWOW2Q, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.700693686 (14h13m28.215796268s) [noise=true , address=104.36.82.46:4515 , piece_id=AMUEDZI5RDYCSKAT6LDWLPBSJ2GCPFM2FCWTESJISYPN3IBC3SCA, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.701810303 (14h13m28.21691288s) [noise=true , address=104.36.82.39:4741 , piece_id=JWBNNZNT3ZOTCDKCV7EG4GECH422LP5CDRHL4CFGTJ2EMTNLJN5A, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.702831457 (14h13m28.217934039s) [noise=true , address=45.91.116.9:4290 , piece_id=TFE3T2MFGXLT4GLD6JBTE73MHIC7QGGNIUWDS7LKM2D4CMQZIOYA, seg_pos={0 0}, upload=14631]: Uploading piece... Aug 7 23:52:53.767754625 (14h13m28.282857196s) [upload=14631]: Next returned nil segment Aug 7 23:52:53.767768780 (14h13m28.282871345s) [upload=14631]: Waiting for error group managing segments... Aug 7 23:52:53.777227926 (14h13m28.292330496s) [noise=true , address=104.36.82.39:4746 , piece_id=NMKV5KBU3V2KVSFRITKGNFPQISKIFK3RDKJUZQSH3MBU26MXHDRA, seg_pos={0 0}, upload=14631]: Done uploading piece. err: <nil> Aug 7 23:52:53.778525320 (14h13m28.293627897s) [noise=true , address=104.36.82.46:4515 , piece_id=AMUEDZI5RDYCSKAT6LDWLPBSJ2GCPFM2FCWTESJISYPN3IBC3SCA, seg_pos={0 0}, upload=14631]: Done uploading piece. err: <nil> Aug 7 23:52:53.788240783 (14h13m28.303343347s) [noise=true , address=104.36.82.16:4296 , piece_id=SPEE6NAFLQF74MDZYA7K6IBVUILHAWOKJSBSGKF6QJTBJ6CGDGBA, seg_pos={0 0}, upload=14631]: Done uploading piece. err: <nil> Aug 7 23:52:53.789097991 (14h13m28.304200562s) [noise=true , address=104.36.82.39:4741 , piece_id=JWBNNZNT3ZOTCDKCV7EG4GECH422LP5CDRHL4CFGTJ2EMTNLJN5A, seg_pos={0 0}, upload=14631]: Done uploading piece. err: <nil> Aug 7 23:52:53.790182855 (14h13m28.305285426s) [noise=true , address=45.91.116.9:4290 , piece_id=TFE3T2MFGXLT4GLD6JBTE73MHIC7QGGNIUWDS7LKM2D4CMQZIOYA, seg_pos={0 0}, upload=14631]: Done uploading piece. err: <nil> Aug 7 23:52:53.791086415 (14h13m28.306188981s) [noise=true , address=45.91.116.4:4964 , piece_id=AAEEGFTLA3USENXJPUS2UD6SW2YTP4GMOAOO4GCBH2AYYT73DFAQ, seg_pos={0 0}, upload=14631]: Done uploading piece. err: <nil> Aug 7 23:52:53.792514550 (14h13m28.307617117s) [noise=true , address=104.36.82.54:4165 , piece_id=KQ776X5RH6VYMMEZ2WGEAHFIQVF6257XLIOOQL7ZPPDBGNUD4PYA, seg_pos={0 0}, upload=14631]: Done uploading piece. err: <nil> Aug 7 23:52:53.806891024 (14h13m28.321993602s) [noise=true , address=104.36.82.52:4098 , piece_id=RXWXDTPQUUCJ57XTCO3TQOBSYDFQ4MZGH7O6E3OFMZULLN3P7ITQ, seg_pos={0 0}, upload=14631]: Done uploading piece. err: <nil> Aug 7 23:52:53.811849722 (14h13m28.326952306s) [noise=true , address=104.36.82.4:4824 , piece_id=F4YOXTKWOGGT5QHYXFIZL4FJOOVHBKEGHHEJXI2ZX2HJ6ZKYN5CQ, seg_pos={0 0}, upload=14631]: Done uploading piece. err: <nil> Aug 7 23:52:53.827462777 (14h13m28.342565345s) [noise=true , address=104.36.82.23:4459 , piece_id=PP6FJQDJ3TMKUYF4LEDPR4JRA5GZ6ZDMC24M6BVTGDH4DFSS6GIQ, seg_pos={0 0}, upload=14631]: Done uploading piece. err: <nil> Aug 7 23:52:53.831869285 (14h13m28.346971865s) [noise=true , address=104.36.82.47:4525 , piece_id=BE22FSLCHMRFEWKAWWBP3I7KUO5PFQGWSJEVTBTFT223GCRWOW2Q, seg_pos={0 0}, upload=14631]: Done uploading piece. err: <nil> Aug 7 23:52:53.837103172 (14h13m28.352205754s) [noise=true , address=104.36.82.9:4028 , piece_id=X63TSF2TVUZXVEAL2RA3IMSXF7VCXHA442QJCCJEMNTC2YIWZMWA, seg_pos={0 0}, upload=14631]: Done uploading piece. err: <nil> Aug 7 23:52:53.982157141 (14h13m28.497259719s) [noise=true , address=104.36.82.44:4898 , piece_id=3PB72NHDN3PIPM4VMYPOELLIMMYOMCM3KOE2A77F4VVYIWQWMNGA, seg_pos={0 0}, upload=14631]: Done uploading piece. err: <nil> Aug 7 23:52:54.036537934 (14h13m28.551640504s) [noise=true , address=79.127.201.215:4356 , piece_id=OSQLCO2P5NKWKD5KGEX2KW6UNC3XUE6KYBDWH47NH52IHG7IXOUQ, seg_pos={0 0}, upload=14631]: Done uploading piece. err: <nil> Aug 7 23:52:54.036561698 (14h13m28.551664274s) [seg_pos={0 0}, upload=14631]: Segment reached optimal threshold of 14 pieces. Aug 7 23:52:54.036864257 (14h13m28.551966838s) [noise=true , address=104.36.82.43:4838 , piece_id=KLNUL263DQQUZ5BJQTBFUZEVYXKVUFMSLHERH6LBNDIXMJWZ23KQ, seg_pos={0 0}, upload=14631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eorQLNmgEaXv8myr6t3Q8rysGwsvtdZzu5b7sndhoANb6Waj6): protocol: expected piece hash; context canceled; EOF Aug 7 23:52:54.037055935 (14h13m28.552158502s) [noise=true , address=104.36.82.10:4109 , piece_id=24NFW3W3KBWD5GPQXNTICBF5YHFVLQLZ7IMJOZNR4DKNTONSY4HA, seg_pos={0 0}, upload=14631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZE1ut9Tum4tzq2m8zbFaGUNLdNZSZQYARkayDAfE1b7uZtWjW): protocol: write tcp 159.223.143.39:48750->104.36.82.10:4109: use of closed network connection; write tcp 159.223.143.39:48750->104.36.82.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48750->104.36.82.10:4109: use of closed network connection Aug 7 23:52:54.037258369 (14h13m28.552360935s) [noise=true , address=104.36.82.35:4590 , piece_id=4USMUKHDNE2WBPE5GGEB3P4SC77CCFLLETXVPPOXJ2ZUNTDER7ZQ, seg_pos={0 0}, upload=14631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128t6B6PhCQznqwMTGJdkWuuCyqQ4bVHbPbZWWnYwb8Eyc1DjDw): protocol: write tcp 159.223.143.39:49450->104.36.82.35:4590: use of closed network connection; write tcp 159.223.143.39:49450->104.36.82.35:4590: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49450->104.36.82.35:4590: use of closed network connection Aug 7 23:52:54.037340084 (14h13m28.552442668s) [noise=true , address=104.36.82.45:4926 , piece_id=PGKQGEMNS2AP423PUETAZZIXNNI7ZTQOWY6SGXDWVQ3DMRQECQGQ, seg_pos={0 0}, upload=14631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LkGN9UdiF163eg78WUPpXqz4FxHBxhnPin4okcdU1YPDWx4MF): protocol: read tcp 159.223.143.39:32804->104.36.82.45:4926: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:52:54.037370062 (14h13m28.552472628s) [seg_pos={0 0}, upload=14631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:52:54.037456923 (14h13m28.552559488s) [upload=14631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:52:54.037461311 (14h13m28.552563877s) [upload=14631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:13.558610345 (14h13m48.07371291s) [upload=14631]: Done uploading segments... Aug 7 23:53:13.558627739 (14h13m48.073730309s) [upload=14631]: Upload finished. err: <nil> Duration: 1m0.002267957s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......XXX..X.X.X.....................................X.....] duration: 611.794425ms FAIL (2025-08-07 23:53:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 611.815817ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X...........................................XX] duration: 30.001420107s FAIL (2025-08-07 23:53:36 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== load=14606]: Uploading piece... Aug 7 23:51:35.405331824 (14h12m9.92043441s) [noise=true , address=45.91.116.4:4939 , piece_id=H6RWITCLY42NEGDEDTBG657Y3PB2OAJQ6KEK7C3WLFLQZRP6WU3Q, seg_pos={0 0}, upload=14606]: Uploading piece... Aug 7 23:51:35.405342181 (14h12m9.920444758s) [noise=true , address=104.36.82.2:4732 , piece_id=2GK4RULPRTBTX7LSEYQUT4UR6B5AYJSYNQAA5YDCCBSIRAUPLTOA, seg_pos={0 0}, upload=14606]: Uploading piece... Aug 7 23:51:35.405694693 (14h12m9.920797273s) [noise=true , address=104.36.82.5:4839 , piece_id=42C3BS7PSBMETSEEP3GWSOH4WUT2JUHH5QGBNZ5TAANW4WDOGH2Q, seg_pos={0 0}, upload=14606]: Uploading piece... Aug 7 23:51:35.405707020 (14h12m9.920809614s) [noise=true , address=45.91.116.12:4225 , piece_id=3223EE4DK2F7PMVC57QVG4IB5SK6VBZOP4OXEIZTAR23DH4N3FJQ, seg_pos={0 0}, upload=14606]: Uploading piece... Aug 7 23:51:35.406015708 (14h12m9.921118287s) [noise=true , address=104.36.82.13:4186 , piece_id=QAOLE7WMNTFWJYCX44POJU2KE4EH65OXAXCFBI2CLP5IHUPPT5VA, seg_pos={0 0}, upload=14606]: Uploading piece... Aug 7 23:51:35.406068948 (14h12m9.921171527s) [noise=true , address=77.90.31.13:4628 , piece_id=4KYOK67JOALTBRXBHBYDWBDG75HZQAZVRWYM2ECKDKZZ3SXKP3XQ, seg_pos={0 0}, upload=14606]: Uploading piece... Aug 7 23:51:35.406318146 (14h12m9.921420725s) [noise=true , address=104.36.82.14:4221 , piece_id=KYCQMYOIN3MV5JW63KCLACFFDQWYMJZTRVEUOVFGCX3XWLRI5DGQ, seg_pos={0 0}, upload=14606]: Uploading piece... Aug 7 23:51:35.406682793 (14h12m9.92178538s) [noise=true , address=104.36.82.8:4971 , piece_id=ZC54WOM3C7LNVM2X6P5I2GPGZ7IZTORQ2DYUYDM7US3GKOYGQRHA, seg_pos={0 0}, upload=14606]: Uploading piece... Aug 7 23:51:35.407100619 (14h12m9.922203195s) [noise=true , address=104.36.82.16:4294 , piece_id=XIHQZDR5IMQQMZTO75Q7KKUM5ZR6532ABE3TI55JB4YALRWF4T5Q, seg_pos={0 0}, upload=14606]: Uploading piece... Aug 7 23:51:35.407521118 (14h12m9.922623705s) [noise=true , address=45.91.116.8:4047 , piece_id=34GCHEBUJ2PX7AJ5A2NQQSGMDX6PNUMTCMCPXTFMFW3GDUKKFWBQ, seg_pos={0 0}, upload=14606]: Uploading piece... Aug 7 23:51:35.408541750 (14h12m9.92364433s) [noise=true , address=104.36.82.15:4258 , piece_id=CE5COWKW32DRRZBBMT4RJKLL3Q6PANJYQDRR524YYWNLFZDE3YVA, seg_pos={0 0}, upload=14606]: Uploading piece... Aug 7 23:51:35.408920069 (14h12m9.924022648s) [noise=true , address=45.91.116.13:4155 , piece_id=3XXELTRHRR5RDDEYHV5TBNEUPUTIY44CKIAH5N7KTTKBZ3GNFYEQ, seg_pos={0 0}, upload=14606]: Uploading piece... Aug 7 23:51:35.409269498 (14h12m9.924372079s) [noise=true , address=104.36.82.12:4151 , piece_id=FQ7RTJ6RSKQXLY2QLWMPZNBFLQOH7RYV2QMDM247YKPOQZ43IJXQ, seg_pos={0 0}, upload=14606]: Uploading piece... Aug 7 23:51:35.412783363 (14h12m9.927885934s) [noise=true , address=77.90.31.14:4661 , piece_id=2UBBTMXSKDSUCA7SCOMGERAWBHDNIMZXP6PTJPZ2WN76KIUGATIA, seg_pos={0 0}, upload=14606]: Uploading piece... Aug 7 23:51:35.524193177 (14h12m10.039295755s) [upload=14606]: Next returned nil segment Aug 7 23:51:35.524218480 (14h12m10.039321059s) [upload=14606]: Waiting for error group managing segments... Aug 7 23:51:35.528680921 (14h12m10.04378349s) [noise=true , address=77.90.31.14:4661 , piece_id=2UBBTMXSKDSUCA7SCOMGERAWBHDNIMZXP6PTJPZ2WN76KIUGATIA, seg_pos={0 0}, upload=14606]: Done uploading piece. err: <nil> Aug 7 23:51:35.529111404 (14h12m10.044213966s) [noise=true , address=45.91.116.8:4047 , piece_id=34GCHEBUJ2PX7AJ5A2NQQSGMDX6PNUMTCMCPXTFMFW3GDUKKFWBQ, seg_pos={0 0}, upload=14606]: Done uploading piece. err: <nil> Aug 7 23:51:35.529204740 (14h12m10.044307308s) [noise=true , address=45.91.116.12:4225 , piece_id=3223EE4DK2F7PMVC57QVG4IB5SK6VBZOP4OXEIZTAR23DH4N3FJQ, seg_pos={0 0}, upload=14606]: Done uploading piece. err: <nil> Aug 7 23:51:35.531697560 (14h12m10.046800142s) [noise=true , address=77.90.31.13:4628 , piece_id=4KYOK67JOALTBRXBHBYDWBDG75HZQAZVRWYM2ECKDKZZ3SXKP3XQ, seg_pos={0 0}, upload=14606]: Done uploading piece. err: <nil> Aug 7 23:51:35.532347107 (14h12m10.047449676s) [noise=true , address=45.91.116.4:4939 , piece_id=H6RWITCLY42NEGDEDTBG657Y3PB2OAJQ6KEK7C3WLFLQZRP6WU3Q, seg_pos={0 0}, upload=14606]: Done uploading piece. err: <nil> Aug 7 23:51:35.532868683 (14h12m10.047971245s) [noise=true , address=45.91.116.13:4155 , piece_id=3XXELTRHRR5RDDEYHV5TBNEUPUTIY44CKIAH5N7KTTKBZ3GNFYEQ, seg_pos={0 0}, upload=14606]: Done uploading piece. err: <nil> Aug 7 23:51:35.533320354 (14h12m10.048422922s) [noise=true , address=77.90.31.11:4555 , piece_id=6T4BNHJ65FRZ75LLIJ6KBKTIJYO7IGFBLRHMHS7U5LYSZ2SRNBTQ, seg_pos={0 0}, upload=14606]: Done uploading piece. err: <nil> Aug 7 23:51:35.538791806 (14h12m10.053894368s) [noise=true , address=104.36.82.15:4259 , piece_id=PGSORNWLXW6RC74ATT3NFSY6VC7W4TFLO2LAMGUYV3RMRVWPMNQQ, seg_pos={0 0}, upload=14606]: Done uploading piece. err: <nil> Aug 7 23:51:35.539514067 (14h12m10.054616646s) [noise=true , address=104.36.82.2:4732 , piece_id=2GK4RULPRTBTX7LSEYQUT4UR6B5AYJSYNQAA5YDCCBSIRAUPLTOA, seg_pos={0 0}, upload=14606]: Done uploading piece. err: <nil> Aug 7 23:51:35.539606529 (14h12m10.054709099s) [noise=true , address=104.36.82.16:4294 , piece_id=XIHQZDR5IMQQMZTO75Q7KKUM5ZR6532ABE3TI55JB4YALRWF4T5Q, seg_pos={0 0}, upload=14606]: Done uploading piece. err: <nil> Aug 7 23:51:35.540330858 (14h12m10.055433492s) [noise=true , address=104.36.82.5:4839 , piece_id=42C3BS7PSBMETSEEP3GWSOH4WUT2JUHH5QGBNZ5TAANW4WDOGH2Q, seg_pos={0 0}, upload=14606]: Done uploading piece. err: <nil> Aug 7 23:51:35.543245378 (14h12m10.058347955s) [noise=true , address=104.36.82.12:4151 , piece_id=FQ7RTJ6RSKQXLY2QLWMPZNBFLQOH7RYV2QMDM247YKPOQZ43IJXQ, seg_pos={0 0}, upload=14606]: Done uploading piece. err: <nil> Aug 7 23:51:35.557579820 (14h12m10.072682407s) [noise=true , address=104.36.82.14:4221 , piece_id=KYCQMYOIN3MV5JW63KCLACFFDQWYMJZTRVEUOVFGCX3XWLRI5DGQ, seg_pos={0 0}, upload=14606]: Done uploading piece. err: <nil> Aug 7 23:51:35.557631566 (14h12m10.072734144s) [seg_pos={0 0}, upload=14606]: Segment reached optimal threshold of 13 pieces. Aug 7 23:51:35.558140786 (14h12m10.073243372s) [noise=true , address=104.36.82.8:4971 , piece_id=ZC54WOM3C7LNVM2X6P5I2GPGZ7IZTORQ2DYUYDM7US3GKOYGQRHA, seg_pos={0 0}, upload=14606]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 159.223.143.39:58716->104.36.82.8:4971: use of closed network connection; write tcp 159.223.143.39:58716->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58716->104.36.82.8:4971: use of closed network connection Aug 7 23:51:35.558325994 (14h12m10.07342858s) [noise=true , address=104.36.82.15:4258 , piece_id=CE5COWKW32DRRZBBMT4RJKLL3Q6PANJYQDRR524YYWNLFZDE3YVA, seg_pos={0 0}, upload=14606]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: read tcp 159.223.143.39:33696->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:51:35.558419532 (14h12m10.0735221s) [noise=true , address=104.36.82.13:4186 , piece_id=QAOLE7WMNTFWJYCX44POJU2KE4EH65OXAXCFBI2CLP5IHUPPT5VA, seg_pos={0 0}, upload=14606]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: read tcp 159.223.143.39:41094->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:51:35.558451109 (14h12m10.073553694s) [seg_pos={0 0}, upload=14606]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:51:35.558580243 (14h12m10.073682807s) [upload=14606]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:51:35.558586543 (14h12m10.073689127s) [upload=14606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:35.707214657 (14h12m10.22231724s) [upload=14606]: Done uploading segments... Aug 7 23:51:35.707291109 (14h12m10.222393688s) [upload=14606]: Upload finished. err: <nil> Aug 7 23:52:35.986308779 (14h13m10.501411356s) [upload=14629]: Starting upload Aug 7 23:52:35.986399932 (14h13m10.501502509s) [upload=14629]: Done starting upload Aug 7 23:52:35.986490908 (14h13m10.501593485s) [upload=14629]: Uploading segments... Aug 7 23:52:35.986775485 (14h13m10.501878054s) [upload=14629]: Got next segment. Inline: false Aug 7 23:52:35.986847602 (14h13m10.501950179s) [upload=14629]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:35.986869666 (14h13m10.501972243s) [upload=14629]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:53:05.954641948 (14h13m40.469744682s) [upload=14629]: Next segment err: context deadline exceeded Aug 7 23:53:05.954726017 (14h13m40.469828594s) [upload=14629]: Done uploading segments... Aug 7 23:53:05.954747075 (14h13m40.469849637s) [upload=14629]: Upload finished. err: context deadline exceeded Aug 7 23:53:06.053410441 (14h13m40.568513021s) [upload=14639]: Starting upload Aug 7 23:53:06.053513846 (14h13m40.568616435s) [upload=14639]: Done starting upload Aug 7 23:53:06.053655272 (14h13m40.568757841s) [upload=14639]: Uploading segments... Aug 7 23:53:06.053907022 (14h13m40.569009591s) [upload=14639]: Got next segment. Inline: false Aug 7 23:53:06.053961798 (14h13m40.569064377s) [upload=14639]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:06.053967636 (14h13m40.569070215s) [upload=14639]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:53:36.020489094 (14h14m10.535591683s) [upload=14639]: Next segment err: context deadline exceeded Aug 7 23:53:36.020570412 (14h14m10.535672979s) [upload=14639]: Done uploading segments... Aug 7 23:53:36.020593471 (14h14m10.535696042s) [upload=14639]: Upload finished. err: context deadline exceeded Duration: 30.001438466s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 6 out of 60 is failed (threshold=3) [......XX.XXX..............................................X.] duration: 225.897243ms FAIL (2025-08-07 23:53:25 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (14h5m24.278312124s) [upload=14493]: Done uploading segments... Aug 7 23:44:49.763230051 (14h5m24.278332629s) [upload=14493]: Upload finished. err: <nil> Aug 7 23:45:27.926377174 (14h6m2.441479751s) [upload=14504]: Starting upload Aug 7 23:45:27.926457453 (14h6m2.441560032s) [upload=14504]: Done starting upload Aug 7 23:45:27.926551181 (14h6m2.441653749s) [upload=14504]: Uploading segments... Aug 7 23:45:27.926604205 (14h6m2.441706791s) [upload=14504]: Got next segment. Inline: true Aug 7 23:45:27.926609625 (14h6m2.441712202s) [upload=14504]: Waiting for error group managing segments... Aug 7 23:45:27.926654474 (14h6m2.441757049s) [upload=14504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:45:27.926672440 (14h6m2.441775019s) [upload=14504]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:45:27.926676591 (14h6m2.441779167s) [upload=14504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:45:28.016681266 (14h6m2.531783833s) [upload=14504]: Done uploading segments... Aug 7 23:45:28.016697602 (14h6m2.531800179s) [upload=14504]: Upload finished. err: <nil> Aug 7 23:45:44.705268158 (14h6m19.220370733s) [upload=14512]: Starting upload Aug 7 23:45:44.705354905 (14h6m19.220457482s) [upload=14512]: Done starting upload Aug 7 23:45:44.705494426 (14h6m19.220596993s) [upload=14512]: Uploading segments... Aug 7 23:45:44.705564460 (14h6m19.220667036s) [upload=14512]: Got next segment. Inline: true Aug 7 23:45:44.705570161 (14h6m19.220672738s) [upload=14512]: Waiting for error group managing segments... Aug 7 23:45:44.705618992 (14h6m19.220721555s) [upload=14512]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:45:44.705641243 (14h6m19.22074382s) [upload=14512]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:45:44.705647157 (14h6m19.220749744s) [upload=14512]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:45:44.774839783 (14h6m19.289942368s) [upload=14512]: Done uploading segments... Aug 7 23:45:44.774865128 (14h6m19.289967705s) [upload=14512]: Upload finished. err: <nil> Aug 7 23:46:55.221569882 (14h7m29.736672449s) [upload=14530]: Starting upload Aug 7 23:46:55.221642729 (14h7m29.736745314s) [upload=14530]: Done starting upload Aug 7 23:46:55.221734906 (14h7m29.736837478s) [upload=14530]: Uploading segments... Aug 7 23:46:55.221804000 (14h7m29.736906588s) [upload=14530]: Got next segment. Inline: true Aug 7 23:46:55.221811160 (14h7m29.736913729s) [upload=14530]: Waiting for error group managing segments... Aug 7 23:46:55.221885036 (14h7m29.7369876s) [upload=14530]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:46:55.221890384 (14h7m29.736992966s) [upload=14530]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:46:55.221893621 (14h7m29.736996201s) [upload=14530]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:46:55.321692569 (14h7m29.836795155s) [upload=14530]: Done uploading segments... Aug 7 23:46:55.321713851 (14h7m29.836816438s) [upload=14530]: Upload finished. err: <nil> Aug 7 23:47:58.537981593 (14h8m33.053084189s) [upload=14545]: Starting upload Aug 7 23:47:58.538091567 (14h8m33.053194153s) [upload=14545]: Done starting upload Aug 7 23:47:58.538175758 (14h8m33.053278336s) [upload=14545]: Uploading segments... Aug 7 23:47:58.538236654 (14h8m33.053339241s) [upload=14545]: Got next segment. Inline: true Aug 7 23:47:58.538243619 (14h8m33.053346194s) [upload=14545]: Waiting for error group managing segments... Aug 7 23:47:58.538296503 (14h8m33.053399195s) [upload=14545]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:47:58.538302975 (14h8m33.053405563s) [upload=14545]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:47:58.538308908 (14h8m33.053411487s) [upload=14545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:47:58.619189912 (14h8m33.134292494s) [upload=14545]: Done uploading segments... Aug 7 23:47:58.619212540 (14h8m33.134315118s) [upload=14545]: Upload finished. err: <nil> Aug 7 23:49:01.871383066 (14h9m36.386485637s) [upload=14563]: Starting upload Aug 7 23:49:01.871540971 (14h9m36.386643551s) [upload=14563]: Done starting upload Aug 7 23:49:01.871671051 (14h9m36.386773634s) [upload=14563]: Uploading segments... Aug 7 23:49:01.871754766 (14h9m36.386857337s) [upload=14563]: Got next segment. Inline: true Aug 7 23:49:01.871764735 (14h9m36.386867325s) [upload=14563]: Waiting for error group managing segments... Aug 7 23:49:01.871838683 (14h9m36.386941251s) [upload=14563]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:49:01.871845530 (14h9m36.386948123s) [upload=14563]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:49:01.871849510 (14h9m36.38695209s) [upload=14563]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:49:01.934052118 (14h9m36.449154695s) [upload=14563]: Done uploading segments... Aug 7 23:49:01.934071674 (14h9m36.44917425s) [upload=14563]: Upload finished. err: <nil> Aug 7 23:49:30.817755150 (14h10m5.332857717s) [upload=14567]: Starting upload Aug 7 23:49:30.817852337 (14h10m5.332954913s) [upload=14567]: Done starting upload Aug 7 23:49:30.817934904 (14h10m5.333037472s) [upload=14567]: Uploading segments... Aug 7 23:49:30.818001589 (14h10m5.333104172s) [upload=14567]: Got next segment. Inline: true Aug 7 23:49:30.818007295 (14h10m5.333109877s) [upload=14567]: Waiting for error group managing segments... Aug 7 23:49:30.818078537 (14h10m5.333181104s) [upload=14567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:49:30.818084049 (14h10m5.333186629s) [upload=14567]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:49:30.818087176 (14h10m5.333189753s) [upload=14567]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:49:30.888782953 (14h10m5.403885532s) [upload=14567]: Done uploading segments... Aug 7 23:49:30.888817710 (14h10m5.403920277s) [upload=14567]: Upload finished. err: <nil> Aug 7 23:50:32.085473195 (14h11m6.600575774s) [upload=14591]: Starting upload Aug 7 23:50:32.085550609 (14h11m6.600653185s) [upload=14591]: Done starting upload Aug 7 23:50:32.085610615 (14h11m6.600713183s) [upload=14591]: Uploading segments... Aug 7 23:50:32.085658436 (14h11m6.600761018s) [upload=14591]: Got next segment. Inline: true Aug 7 23:50:32.085662726 (14h11m6.600765303s) [upload=14591]: Waiting for error group managing segments... Aug 7 23:50:32.085716357 (14h11m6.600818923s) [upload=14591]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:32.085721348 (14h11m6.600823924s) [upload=14591]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:32.085724995 (14h11m6.600827572s) [upload=14591]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:50:32.142060473 (14h11m6.657163039s) [upload=14591]: Done uploading segments... Aug 7 23:50:32.142076779 (14h11m6.657179345s) [upload=14591]: Upload finished. err: <nil> Aug 7 23:50:57.356595368 (14h11m31.871697934s) [upload=14595]: Starting upload Aug 7 23:50:57.356765546 (14h11m31.871868133s) [upload=14595]: Done starting upload Aug 7 23:50:57.356869337 (14h11m31.871971899s) [upload=14595]: Uploading segments... Aug 7 23:50:57.356914930 (14h11m31.872017522s) [upload=14595]: Got next segment. Inline: true Aug 7 23:50:57.356921513 (14h11m31.872024098s) [upload=14595]: Waiting for error group managing segments... Aug 7 23:50:57.356982351 (14h11m31.872084938s) [upload=14595]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:50:57.356989548 (14h11m31.872092141s) [upload=14595]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:50:57.356995954 (14h11m31.872098543s) [upload=14595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:04.638440451 (14h11m39.153543026s) [upload=14595]: Done uploading segments... Aug 7 23:51:04.638457479 (14h11m39.153560045s) [upload=14595]: Upload finished. err: <nil> Aug 7 23:52:06.228205136 (14h12m40.743307702s) [upload=14620]: Starting upload Aug 7 23:52:06.228273353 (14h12m40.74337593s) [upload=14620]: Done starting upload Aug 7 23:52:06.228361368 (14h12m40.743463938s) [upload=14620]: Uploading segments... Aug 7 23:52:06.228433067 (14h12m40.743535646s) [upload=14620]: Got next segment. Inline: true Aug 7 23:52:06.228437780 (14h12m40.743540356s) [upload=14620]: Waiting for error group managing segments... Aug 7 23:52:06.228479528 (14h12m40.743582104s) [upload=14620]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:06.228483802 (14h12m40.743586378s) [upload=14620]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:06.228498744 (14h12m40.743601323s) [upload=14620]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:06.292782592 (14h12m40.807885172s) [upload=14620]: Done uploading segments... Aug 7 23:52:06.292803751 (14h12m40.80790633s) [upload=14620]: Upload finished. err: <nil> Aug 7 23:52:55.204638563 (14h13m29.719741157s) [upload=14636]: Starting upload Aug 7 23:52:55.204832733 (14h13m29.719935326s) [upload=14636]: Done starting upload Aug 7 23:52:55.204913770 (14h13m29.720016348s) [upload=14636]: Uploading segments... Aug 7 23:52:55.205057568 (14h13m29.720160151s) [upload=14636]: Got next segment. Inline: true Aug 7 23:52:55.205064394 (14h13m29.720166971s) [upload=14636]: Waiting for error group managing segments... Aug 7 23:52:55.205114279 (14h13m29.720216864s) [upload=14636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:55.205134068 (14h13m29.720236652s) [upload=14636]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:55.205137808 (14h13m29.72024039s) [upload=14636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:53:25.172932226 (14h13m59.688034802s) [upload=14636]: Done uploading segments... Aug 7 23:53:25.172962711 (14h13m59.688065287s) [upload=14636]: Upload finished. err: metaclient: context deadline exceeded Duration: 225.918026ms core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 42.305477ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 23:53:25 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 42.332307ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 31.674836ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 23:53:25 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 31.693066ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [........XXX..X.X.X.....................................X....] duration: 610.636977ms FAIL (2025-08-07 23:53:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 610.663151ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [.....XX.X.X....XX..........................................X] duration: 30.000517142s FAIL (2025-08-07 23:53:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000538554s edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 27.340557ms target: https://us1.storj.io/login FAIL (2025-08-07 23:53:09 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 27.362034ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X...........................................X] duration: 30.000664153s FAIL (2025-08-07 23:53:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 5614202 (14h12m9.840716784s) [upload=14606]: Got next segment. Inline: false Aug 7 23:51:35.325688584 (14h12m9.840791161s) [upload=14606]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:51:35.325693405 (14h12m9.840795972s) [upload=14606]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:51:35.404828280 (14h12m9.919930869s) [seg_pos={0 0}, upload=14606]: Begin upload segment... Aug 7 23:51:35.404880268 (14h12m9.919982845s) [seg_pos={0 0}, upload=14606]: Done begin upload segment. Aug 7 23:51:35.404913210 (14h12m9.920015794s) [noise=true , address=104.36.82.15:4259 , piece_id=PGSORNWLXW6RC74ATT3NFSY6VC7W4TFLO2LAMGUYV3RMRVWPMNQQ, seg_pos={0 0}, upload=14606]: Uploading piece... Aug 7 23:51:35.404932311 (14h12m9.920034894s) [noise=true , address=77.90.31.11:4555 , piece_id=6T4BNHJ65FRZ75LLIJ6KBKTIJYO7IGFBLRHMHS7U5LYSZ2SRNBTQ, seg_pos={0 0}, upload=14606]: Uploading piece... Aug 7 23:51:35.405331824 (14h12m9.92043441s) [noise=true , address=45.91.116.4:4939 , piece_id=H6RWITCLY42NEGDEDTBG657Y3PB2OAJQ6KEK7C3WLFLQZRP6WU3Q, seg_pos={0 0}, upload=14606]: Uploading piece... Aug 7 23:51:35.405342181 (14h12m9.920444758s) [noise=true , address=104.36.82.2:4732 , piece_id=2GK4RULPRTBTX7LSEYQUT4UR6B5AYJSYNQAA5YDCCBSIRAUPLTOA, seg_pos={0 0}, upload=14606]: Uploading piece... Aug 7 23:51:35.405694693 (14h12m9.920797273s) [noise=true , address=104.36.82.5:4839 , piece_id=42C3BS7PSBMETSEEP3GWSOH4WUT2JUHH5QGBNZ5TAANW4WDOGH2Q, seg_pos={0 0}, upload=14606]: Uploading piece... Aug 7 23:51:35.405707020 (14h12m9.920809614s) [noise=true , address=45.91.116.12:4225 , piece_id=3223EE4DK2F7PMVC57QVG4IB5SK6VBZOP4OXEIZTAR23DH4N3FJQ, seg_pos={0 0}, upload=14606]: Uploading piece... Aug 7 23:51:35.406015708 (14h12m9.921118287s) [noise=true , address=104.36.82.13:4186 , piece_id=QAOLE7WMNTFWJYCX44POJU2KE4EH65OXAXCFBI2CLP5IHUPPT5VA, seg_pos={0 0}, upload=14606]: Uploading piece... Aug 7 23:51:35.406068948 (14h12m9.921171527s) [noise=true , address=77.90.31.13:4628 , piece_id=4KYOK67JOALTBRXBHBYDWBDG75HZQAZVRWYM2ECKDKZZ3SXKP3XQ, seg_pos={0 0}, upload=14606]: Uploading piece... Aug 7 23:51:35.406318146 (14h12m9.921420725s) [noise=true , address=104.36.82.14:4221 , piece_id=KYCQMYOIN3MV5JW63KCLACFFDQWYMJZTRVEUOVFGCX3XWLRI5DGQ, seg_pos={0 0}, upload=14606]: Uploading piece... Aug 7 23:51:35.406682793 (14h12m9.92178538s) [noise=true , address=104.36.82.8:4971 , piece_id=ZC54WOM3C7LNVM2X6P5I2GPGZ7IZTORQ2DYUYDM7US3GKOYGQRHA, seg_pos={0 0}, upload=14606]: Uploading piece... Aug 7 23:51:35.407100619 (14h12m9.922203195s) [noise=true , address=104.36.82.16:4294 , piece_id=XIHQZDR5IMQQMZTO75Q7KKUM5ZR6532ABE3TI55JB4YALRWF4T5Q, seg_pos={0 0}, upload=14606]: Uploading piece... Aug 7 23:51:35.407521118 (14h12m9.922623705s) [noise=true , address=45.91.116.8:4047 , piece_id=34GCHEBUJ2PX7AJ5A2NQQSGMDX6PNUMTCMCPXTFMFW3GDUKKFWBQ, seg_pos={0 0}, upload=14606]: Uploading piece... Aug 7 23:51:35.408541750 (14h12m9.92364433s) [noise=true , address=104.36.82.15:4258 , piece_id=CE5COWKW32DRRZBBMT4RJKLL3Q6PANJYQDRR524YYWNLFZDE3YVA, seg_pos={0 0}, upload=14606]: Uploading piece... Aug 7 23:51:35.408920069 (14h12m9.924022648s) [noise=true , address=45.91.116.13:4155 , piece_id=3XXELTRHRR5RDDEYHV5TBNEUPUTIY44CKIAH5N7KTTKBZ3GNFYEQ, seg_pos={0 0}, upload=14606]: Uploading piece... Aug 7 23:51:35.409269498 (14h12m9.924372079s) [noise=true , address=104.36.82.12:4151 , piece_id=FQ7RTJ6RSKQXLY2QLWMPZNBFLQOH7RYV2QMDM247YKPOQZ43IJXQ, seg_pos={0 0}, upload=14606]: Uploading piece... Aug 7 23:51:35.412783363 (14h12m9.927885934s) [noise=true , address=77.90.31.14:4661 , piece_id=2UBBTMXSKDSUCA7SCOMGERAWBHDNIMZXP6PTJPZ2WN76KIUGATIA, seg_pos={0 0}, upload=14606]: Uploading piece... Aug 7 23:51:35.524193177 (14h12m10.039295755s) [upload=14606]: Next returned nil segment Aug 7 23:51:35.524218480 (14h12m10.039321059s) [upload=14606]: Waiting for error group managing segments... Aug 7 23:51:35.528680921 (14h12m10.04378349s) [noise=true , address=77.90.31.14:4661 , piece_id=2UBBTMXSKDSUCA7SCOMGERAWBHDNIMZXP6PTJPZ2WN76KIUGATIA, seg_pos={0 0}, upload=14606]: Done uploading piece. err: <nil> Aug 7 23:51:35.529111404 (14h12m10.044213966s) [noise=true , address=45.91.116.8:4047 , piece_id=34GCHEBUJ2PX7AJ5A2NQQSGMDX6PNUMTCMCPXTFMFW3GDUKKFWBQ, seg_pos={0 0}, upload=14606]: Done uploading piece. err: <nil> Aug 7 23:51:35.529204740 (14h12m10.044307308s) [noise=true , address=45.91.116.12:4225 , piece_id=3223EE4DK2F7PMVC57QVG4IB5SK6VBZOP4OXEIZTAR23DH4N3FJQ, seg_pos={0 0}, upload=14606]: Done uploading piece. err: <nil> Aug 7 23:51:35.531697560 (14h12m10.046800142s) [noise=true , address=77.90.31.13:4628 , piece_id=4KYOK67JOALTBRXBHBYDWBDG75HZQAZVRWYM2ECKDKZZ3SXKP3XQ, seg_pos={0 0}, upload=14606]: Done uploading piece. err: <nil> Aug 7 23:51:35.532347107 (14h12m10.047449676s) [noise=true , address=45.91.116.4:4939 , piece_id=H6RWITCLY42NEGDEDTBG657Y3PB2OAJQ6KEK7C3WLFLQZRP6WU3Q, seg_pos={0 0}, upload=14606]: Done uploading piece. err: <nil> Aug 7 23:51:35.532868683 (14h12m10.047971245s) [noise=true , address=45.91.116.13:4155 , piece_id=3XXELTRHRR5RDDEYHV5TBNEUPUTIY44CKIAH5N7KTTKBZ3GNFYEQ, seg_pos={0 0}, upload=14606]: Done uploading piece. err: <nil> Aug 7 23:51:35.533320354 (14h12m10.048422922s) [noise=true , address=77.90.31.11:4555 , piece_id=6T4BNHJ65FRZ75LLIJ6KBKTIJYO7IGFBLRHMHS7U5LYSZ2SRNBTQ, seg_pos={0 0}, upload=14606]: Done uploading piece. err: <nil> Aug 7 23:51:35.538791806 (14h12m10.053894368s) [noise=true , address=104.36.82.15:4259 , piece_id=PGSORNWLXW6RC74ATT3NFSY6VC7W4TFLO2LAMGUYV3RMRVWPMNQQ, seg_pos={0 0}, upload=14606]: Done uploading piece. err: <nil> Aug 7 23:51:35.539514067 (14h12m10.054616646s) [noise=true , address=104.36.82.2:4732 , piece_id=2GK4RULPRTBTX7LSEYQUT4UR6B5AYJSYNQAA5YDCCBSIRAUPLTOA, seg_pos={0 0}, upload=14606]: Done uploading piece. err: <nil> Aug 7 23:51:35.539606529 (14h12m10.054709099s) [noise=true , address=104.36.82.16:4294 , piece_id=XIHQZDR5IMQQMZTO75Q7KKUM5ZR6532ABE3TI55JB4YALRWF4T5Q, seg_pos={0 0}, upload=14606]: Done uploading piece. err: <nil> Aug 7 23:51:35.540330858 (14h12m10.055433492s) [noise=true , address=104.36.82.5:4839 , piece_id=42C3BS7PSBMETSEEP3GWSOH4WUT2JUHH5QGBNZ5TAANW4WDOGH2Q, seg_pos={0 0}, upload=14606]: Done uploading piece. err: <nil> Aug 7 23:51:35.543245378 (14h12m10.058347955s) [noise=true , address=104.36.82.12:4151 , piece_id=FQ7RTJ6RSKQXLY2QLWMPZNBFLQOH7RYV2QMDM247YKPOQZ43IJXQ, seg_pos={0 0}, upload=14606]: Done uploading piece. err: <nil> Aug 7 23:51:35.557579820 (14h12m10.072682407s) [noise=true , address=104.36.82.14:4221 , piece_id=KYCQMYOIN3MV5JW63KCLACFFDQWYMJZTRVEUOVFGCX3XWLRI5DGQ, seg_pos={0 0}, upload=14606]: Done uploading piece. err: <nil> Aug 7 23:51:35.557631566 (14h12m10.072734144s) [seg_pos={0 0}, upload=14606]: Segment reached optimal threshold of 13 pieces. Aug 7 23:51:35.558140786 (14h12m10.073243372s) [noise=true , address=104.36.82.8:4971 , piece_id=ZC54WOM3C7LNVM2X6P5I2GPGZ7IZTORQ2DYUYDM7US3GKOYGQRHA, seg_pos={0 0}, upload=14606]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 159.223.143.39:58716->104.36.82.8:4971: use of closed network connection; write tcp 159.223.143.39:58716->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:58716->104.36.82.8:4971: use of closed network connection Aug 7 23:51:35.558325994 (14h12m10.07342858s) [noise=true , address=104.36.82.15:4258 , piece_id=CE5COWKW32DRRZBBMT4RJKLL3Q6PANJYQDRR524YYWNLFZDE3YVA, seg_pos={0 0}, upload=14606]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: read tcp 159.223.143.39:33696->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:51:35.558419532 (14h12m10.0735221s) [noise=true , address=104.36.82.13:4186 , piece_id=QAOLE7WMNTFWJYCX44POJU2KE4EH65OXAXCFBI2CLP5IHUPPT5VA, seg_pos={0 0}, upload=14606]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: read tcp 159.223.143.39:41094->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:51:35.558451109 (14h12m10.073553694s) [seg_pos={0 0}, upload=14606]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:51:35.558580243 (14h12m10.073682807s) [upload=14606]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:51:35.558586543 (14h12m10.073689127s) [upload=14606]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:51:35.707214657 (14h12m10.22231724s) [upload=14606]: Done uploading segments... Aug 7 23:51:35.707291109 (14h12m10.222393688s) [upload=14606]: Upload finished. err: <nil> Aug 7 23:52:35.986308779 (14h13m10.501411356s) [upload=14629]: Starting upload Aug 7 23:52:35.986399932 (14h13m10.501502509s) [upload=14629]: Done starting upload Aug 7 23:52:35.986490908 (14h13m10.501593485s) [upload=14629]: Uploading segments... Aug 7 23:52:35.986775485 (14h13m10.501878054s) [upload=14629]: Got next segment. Inline: false Aug 7 23:52:35.986847602 (14h13m10.501950179s) [upload=14629]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:35.986869666 (14h13m10.501972243s) [upload=14629]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:53:05.954641948 (14h13m40.469744682s) [upload=14629]: Next segment err: context deadline exceeded Aug 7 23:53:05.954726017 (14h13m40.469828594s) [upload=14629]: Done uploading segments... Aug 7 23:53:05.954747075 (14h13m40.469849637s) [upload=14629]: Upload finished. err: context deadline exceeded Duration: 30.000852706s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........XXX..X.X.X.....................................X...] duration: 683.168068ms FAIL (2025-08-07 23:52:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 683.191444ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 9 out of 60 is failed (threshold=3) [....X..XXXXX.X.XX...........................................] duration: 417.07323ms FAIL (2025-08-07 23:52:55 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 417.100507ms core uplink us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 32.046721ms target: https://us1.storj.io/login FAIL (2025-08-07 23:52:44 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 32.08637ms core satellite us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 83.211755ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 23:52:38 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 83.259608ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.....X.XXX.XX...............................................] duration: 812.704353ms FAIL (2025-08-07 23:52:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 812.727783ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 9 out of 60 is failed (threshold=3) [.....X..XXXXX.X.XX..........................................] duration: 255.592104ms FAIL (2025-08-07 23:52:21 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 255.616121ms core uplink us1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 46.962911ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 23:52:20 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 46.988395ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........XXX..X.X.X.....................................X..] duration: 504.683808ms FAIL (2025-08-07 23:52:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 504.733685ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [......XX.X.X....XX..........................................] duration: 322.003489ms FAIL (2025-08-07 23:52:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 322.55084ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 5 out of 60 is failed (threshold=3) [........XX.XXX..............................................] duration: 211.889623ms FAIL (2025-08-07 23:52:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 30m31.135428317s) [upload=13902]: Starting upload Aug 7 23:09:56.620515140 (13h30m31.135617722s) [upload=13902]: Done starting upload Aug 7 23:09:56.620600541 (13h30m31.135703106s) [upload=13902]: Uploading segments... Aug 7 23:09:56.620663979 (13h30m31.135766559s) [upload=13902]: Got next segment. Inline: true Aug 7 23:09:56.620669313 (13h30m31.13577189s) [upload=13902]: Waiting for error group managing segments... Aug 7 23:09:56.620725144 (13h30m31.13582772s) [upload=13902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:56.620729700 (13h30m31.135832276s) [upload=13902]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:56.620732858 (13h30m31.135835435s) [upload=13902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:56.756931332 (13h30m31.272033912s) [upload=13902]: Done uploading segments... Aug 7 23:09:56.756952203 (13h30m31.272054766s) [upload=13902]: Upload finished. err: <nil> Aug 7 23:10:22.255312070 (13h30m56.770414636s) [upload=13913]: Starting upload Aug 7 23:10:22.255389033 (13h30m56.770491609s) [upload=13913]: Done starting upload Aug 7 23:10:22.255538785 (13h30m56.77064136s) [upload=13913]: Uploading segments... Aug 7 23:10:22.255632108 (13h30m56.77073469s) [upload=13913]: Got next segment. Inline: true Aug 7 23:10:22.255639107 (13h30m56.770741683s) [upload=13913]: Waiting for error group managing segments... Aug 7 23:10:22.255686584 (13h30m56.77078915s) [upload=13913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:22.255691754 (13h30m56.770794331s) [upload=13913]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:22.255695470 (13h30m56.770798046s) [upload=13913]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:22.322045125 (13h30m56.837147702s) [upload=13913]: Done uploading segments... Aug 7 23:10:22.322068753 (13h30m56.837171329s) [upload=13913]: Upload finished. err: <nil> Aug 7 23:11:00.342239204 (13h31m34.85734177s) [upload=13922]: Starting upload Aug 7 23:11:00.342310667 (13h31m34.857413235s) [upload=13922]: Done starting upload Aug 7 23:11:00.342383830 (13h31m34.85748641s) [upload=13922]: Uploading segments... Aug 7 23:11:00.342449267 (13h31m34.857551854s) [upload=13922]: Got next segment. Inline: true Aug 7 23:11:00.342455504 (13h31m34.857558092s) [upload=13922]: Waiting for error group managing segments... Aug 7 23:11:00.342503627 (13h31m34.857606209s) [upload=13922]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:00.342508465 (13h31m34.857611046s) [upload=13922]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:00.342511859 (13h31m34.857614451s) [upload=13922]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:00.401913127 (13h31m34.917015712s) [upload=13922]: Done uploading segments... Aug 7 23:11:00.401931904 (13h31m34.917034472s) [upload=13922]: Upload finished. err: <nil> Aug 7 23:11:35.523175401 (13h32m10.03827798s) [upload=13934]: Starting upload Aug 7 23:11:35.523266566 (13h32m10.038369147s) [upload=13934]: Done starting upload Aug 7 23:11:35.523355068 (13h32m10.038457647s) [upload=13934]: Uploading segments... Aug 7 23:11:35.523405664 (13h32m10.03850825s) [upload=13934]: Got next segment. Inline: true Aug 7 23:11:35.523412393 (13h32m10.038514977s) [upload=13934]: Waiting for error group managing segments... Aug 7 23:11:35.523470368 (13h32m10.038572947s) [upload=13934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:35.523479499 (13h32m10.038582083s) [upload=13934]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:35.523484240 (13h32m10.038586821s) [upload=13934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:35.587656651 (13h32m10.102759216s) [upload=13934]: Done uploading segments... Aug 7 23:11:35.587674771 (13h32m10.102777338s) [upload=13934]: Upload finished. err: <nil> Aug 7 23:12:23.226340957 (13h32m57.741443533s) [upload=13947]: Starting upload Aug 7 23:12:23.226411041 (13h32m57.741513618s) [upload=13947]: Done starting upload Aug 7 23:12:23.226470575 (13h32m57.74157314s) [upload=13947]: Uploading segments... Aug 7 23:12:23.226506745 (13h32m57.741609326s) [upload=13947]: Got next segment. Inline: true Aug 7 23:12:23.226511437 (13h32m57.741614019s) [upload=13947]: Waiting for error group managing segments... Aug 7 23:12:23.226561258 (13h32m57.741663839s) [upload=13947]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:23.226566182 (13h32m57.741668762s) [upload=13947]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:23.226569380 (13h32m57.741671956s) [upload=13947]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:53.198277272 (13h33m27.713379848s) [upload=13947]: Done uploading segments... Aug 7 23:12:53.198295252 (13h33m27.713397814s) [upload=13947]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:12:53.310207725 (13h33m27.825310302s) [upload=13957]: Starting upload Aug 7 23:12:53.310290257 (13h33m27.82539282s) [upload=13957]: Done starting upload Aug 7 23:12:53.310379699 (13h33m27.825482276s) [upload=13957]: Uploading segments... Aug 7 23:12:53.310488747 (13h33m27.825591336s) [upload=13957]: Got next segment. Inline: true Aug 7 23:12:53.310497967 (13h33m27.825600548s) [upload=13957]: Waiting for error group managing segments... Aug 7 23:12:53.310564975 (13h33m27.825667562s) [upload=13957]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:53.310572430 (13h33m27.825675017s) [upload=13957]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:53.310576051 (13h33m27.825678628s) [upload=13957]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:23.280527765 (13h33m57.79563034s) [upload=13957]: Done uploading segments... Aug 7 23:13:23.280561185 (13h33m57.795663762s) [upload=13957]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:50.810735096 (13h34m25.325837673s) [upload=13972]: Starting upload Aug 7 23:13:50.810789852 (13h34m25.325892437s) [upload=13972]: Done starting upload Aug 7 23:13:50.810850987 (13h34m25.325953556s) [upload=13972]: Uploading segments... Aug 7 23:13:50.810923216 (13h34m25.326025797s) [upload=13972]: Got next segment. Inline: true Aug 7 23:13:50.810928523 (13h34m25.326031096s) [upload=13972]: Waiting for error group managing segments... Aug 7 23:13:50.810970440 (13h34m25.326073016s) [upload=13972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:50.810974710 (13h34m25.326077286s) [upload=13972]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:50.810978274 (13h34m25.32608085s) [upload=13972]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:50.866577223 (13h34m25.3816798s) [upload=13972]: Done uploading segments... Aug 7 23:13:50.866594654 (13h34m25.38169723s) [upload=13972]: Upload finished. err: <nil> Aug 7 23:14:54.225692687 (13h35m28.740795254s) [upload=13986]: Starting upload Aug 7 23:14:54.225785134 (13h35m28.740887721s) [upload=13986]: Done starting upload Aug 7 23:14:54.225870963 (13h35m28.740973529s) [upload=13986]: Uploading segments... Aug 7 23:14:54.225923809 (13h35m28.741026386s) [upload=13986]: Got next segment. Inline: true Aug 7 23:14:54.225928631 (13h35m28.741031207s) [upload=13986]: Waiting for error group managing segments... Aug 7 23:14:54.225969273 (13h35m28.741071842s) [upload=13986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:54.225973734 (13h35m28.7410763s) [upload=13986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:54.225976991 (13h35m28.741079566s) [upload=13986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:24.191802298 (13h35m58.706904867s) [upload=13986]: Done uploading segments... Aug 7 23:15:24.191822370 (13h35m58.70692494s) [upload=13986]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:24.446619359 (13h35m58.961721924s) [upload=13992]: Starting upload Aug 7 23:15:24.446725754 (13h35m58.961828338s) [upload=13992]: Done starting upload Aug 7 23:15:24.446812887 (13h35m58.961915463s) [upload=13992]: Uploading segments... Aug 7 23:15:24.446871932 (13h35m58.961974519s) [upload=13992]: Got next segment. Inline: true Aug 7 23:15:24.446878746 (13h35m58.961981315s) [upload=13992]: Waiting for error group managing segments... Aug 7 23:15:24.446939653 (13h35m58.962042222s) [upload=13992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:24.446945288 (13h35m58.962047863s) [upload=13992]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:24.446950467 (13h35m58.96205303s) [upload=13992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:54.414392139 (13h36m28.929494705s) [upload=13992]: Done uploading segments... Aug 7 23:15:54.414412085 (13h36m28.929514651s) [upload=13992]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:32.150263694 (13h37m6.665366261s) [upload=14009]: Starting upload Aug 7 23:16:32.150345034 (13h37m6.665447612s) [upload=14009]: Done starting upload Aug 7 23:16:32.150402507 (13h37m6.665505071s) [upload=14009]: Uploading segments... Aug 7 23:16:32.150453597 (13h37m6.665556176s) [upload=14009]: Got next segment. Inline: true Aug 7 23:16:32.150458385 (13h37m6.665560958s) [upload=14009]: Waiting for error group managing segments... Aug 7 23:16:32.150501065 (13h37m6.665603641s) [upload=14009]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:32.150505833 (13h37m6.66560841s) [upload=14009]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:32.150509597 (13h37m6.665612174s) [upload=14009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:02.044102103 (13h37m36.55920467s) [upload=14009]: Done uploading segments... Aug 7 23:17:02.044123177 (13h37m36.559225743s) [upload=14009]: Upload finished. err: metaclient: context deadline exceeded Duration: 211.920333ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 4 out of 60 is failed (threshold=3) [.....XX..X...X..............................................] duration: 209.429348ms FAIL (2025-08-07 23:52:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 162725354 (13h31m20.67782793s) [upload=13919]: Starting upload Aug 7 23:10:46.162807728 (13h31m20.677910304s) [upload=13919]: Done starting upload Aug 7 23:10:46.162877771 (13h31m20.677980347s) [upload=13919]: Uploading segments... Aug 7 23:10:46.162939319 (13h31m20.678041901s) [upload=13919]: Got next segment. Inline: true Aug 7 23:10:46.162944044 (13h31m20.678046624s) [upload=13919]: Waiting for error group managing segments... Aug 7 23:10:46.162986704 (13h31m20.678089286s) [upload=13919]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:46.162991049 (13h31m20.678093629s) [upload=13919]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:46.162994213 (13h31m20.678096792s) [upload=13919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:49.331481110 (13h31m23.84658369s) [upload=13919]: Done uploading segments... Aug 7 23:10:49.331502710 (13h31m23.846605285s) [upload=13919]: Upload finished. err: <nil> Aug 7 23:11:56.588799333 (13h32m31.103901909s) [upload=13942]: Starting upload Aug 7 23:11:56.588871172 (13h32m31.103973747s) [upload=13942]: Done starting upload Aug 7 23:11:56.588959193 (13h32m31.104061755s) [upload=13942]: Uploading segments... Aug 7 23:11:56.588996486 (13h32m31.104099072s) [upload=13942]: Got next segment. Inline: true Aug 7 23:11:56.589000914 (13h32m31.104103491s) [upload=13942]: Waiting for error group managing segments... Aug 7 23:11:56.589047870 (13h32m31.104150433s) [upload=13942]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:56.589051856 (13h32m31.104154432s) [upload=13942]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:56.589054372 (13h32m31.10415695s) [upload=13942]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:56.555383755 (13h33m31.070486344s) [upload=13942]: Done uploading segments... Aug 7 23:12:56.555407795 (13h33m31.070510378s) [upload=13942]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:12:56.669023639 (13h33m31.184126201s) [upload=13959]: Starting upload Aug 7 23:12:56.669081512 (13h33m31.184184088s) [upload=13959]: Done starting upload Aug 7 23:12:56.669139265 (13h33m31.184241827s) [upload=13959]: Uploading segments... Aug 7 23:12:56.669181725 (13h33m31.184284307s) [upload=13959]: Got next segment. Inline: true Aug 7 23:12:56.669185727 (13h33m31.18428829s) [upload=13959]: Waiting for error group managing segments... Aug 7 23:12:56.669222629 (13h33m31.184325191s) [upload=13959]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:56.669226085 (13h33m31.18432866s) [upload=13959]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:56.669228556 (13h33m31.184331118s) [upload=13959]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.639927224 (13h34m31.155029791s) [upload=13959]: Done uploading segments... Aug 7 23:13:56.639946863 (13h34m31.155049429s) [upload=13959]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:56.754611797 (13h34m31.269714367s) [upload=13974]: Starting upload Aug 7 23:13:56.754690544 (13h34m31.269793123s) [upload=13974]: Done starting upload Aug 7 23:13:56.754787910 (13h34m31.269890478s) [upload=13974]: Uploading segments... Aug 7 23:13:56.754829279 (13h34m31.269931864s) [upload=13974]: Got next segment. Inline: true Aug 7 23:13:56.754833834 (13h34m31.269936411s) [upload=13974]: Waiting for error group managing segments... Aug 7 23:13:56.754887650 (13h34m31.269990217s) [upload=13974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:56.754891993 (13h34m31.26999456s) [upload=13974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:56.754895495 (13h34m31.269998071s) [upload=13974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.814228870 (13h34m31.32933145s) [upload=13974]: Done uploading segments... Aug 7 23:13:56.814253732 (13h34m31.329356295s) [upload=13974]: Upload finished. err: <nil> Aug 7 23:14:51.237083244 (13h35m25.75218581s) [upload=13984]: Starting upload Aug 7 23:14:51.237154440 (13h35m25.752257016s) [upload=13984]: Done starting upload Aug 7 23:14:51.237222602 (13h35m25.752325168s) [upload=13984]: Uploading segments... Aug 7 23:14:51.237272573 (13h35m25.752375158s) [upload=13984]: Got next segment. Inline: true Aug 7 23:14:51.237277088 (13h35m25.752379654s) [upload=13984]: Waiting for error group managing segments... Aug 7 23:14:51.237328372 (13h35m25.752430937s) [upload=13984]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:51.237332974 (13h35m25.75243555s) [upload=13984]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:51.237336117 (13h35m25.752438684s) [upload=13984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:51.367347705 (13h35m25.882450281s) [upload=13984]: Done uploading segments... Aug 7 23:14:51.367368302 (13h35m25.882470865s) [upload=13984]: Upload finished. err: <nil> Aug 7 23:16:05.237199068 (13h36m39.752301655s) [upload=14002]: Starting upload Aug 7 23:16:05.237269888 (13h36m39.752372457s) [upload=14002]: Done starting upload Aug 7 23:16:05.237360382 (13h36m39.752462952s) [upload=14002]: Uploading segments... Aug 7 23:16:05.237841750 (13h36m39.752944336s) [upload=14002]: Got next segment. Inline: true Aug 7 23:16:05.237859632 (13h36m39.752962218s) [upload=14002]: Waiting for error group managing segments... Aug 7 23:16:05.237930004 (13h36m39.753032586s) [upload=14002]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:05.237936533 (13h36m39.753039109s) [upload=14002]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:05.237940573 (13h36m39.753043148s) [upload=14002]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:05.205359206 (13h37m39.720461782s) [upload=14002]: Done uploading segments... Aug 7 23:17:05.205376470 (13h37m39.720479046s) [upload=14002]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:15.921193047 (13h37m50.436295623s) [upload=14020]: Starting upload Aug 7 23:17:15.921258731 (13h37m50.436361294s) [upload=14020]: Done starting upload Aug 7 23:17:15.921299906 (13h37m50.436402469s) [upload=14020]: Uploading segments... Aug 7 23:17:15.921331213 (13h37m50.436433799s) [upload=14020]: Got next segment. Inline: true Aug 7 23:17:15.921334685 (13h37m50.436437248s) [upload=14020]: Waiting for error group managing segments... Aug 7 23:17:15.921366597 (13h37m50.436469172s) [upload=14020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:15.921370316 (13h37m50.436472878s) [upload=14020]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:15.921372605 (13h37m50.436475167s) [upload=14020]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:15.996901858 (13h37m50.512004437s) [upload=14020]: Done uploading segments... Aug 7 23:17:15.996920869 (13h37m50.512023431s) [upload=14020]: Upload finished. err: <nil> Aug 7 23:18:10.846312036 (13h38m45.361414612s) [upload=14034]: Starting upload Aug 7 23:18:10.846377244 (13h38m45.36147981s) [upload=14034]: Done starting upload Aug 7 23:18:10.846461339 (13h38m45.361563904s) [upload=14034]: Uploading segments... Aug 7 23:18:10.846496473 (13h38m45.361599058s) [upload=14034]: Got next segment. Inline: true Aug 7 23:18:10.846501164 (13h38m45.36160374s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:18:10.846554786 (13h38m45.361657362s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:10.846559294 (13h38m45.361661871s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:10.846562922 (13h38m45.361665498s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:11.200018957 (13h38m45.715121534s) [upload=14034]: Done uploading segments... Aug 7 23:18:11.200049582 (13h38m45.715152158s) [upload=14034]: Upload finished. err: <nil> Aug 7 23:19:15.896167300 (13h39m50.411269878s) [upload=14051]: Starting upload Aug 7 23:19:15.896353995 (13h39m50.411456637s) [upload=14051]: Done starting upload Aug 7 23:19:15.896436268 (13h39m50.411538832s) [upload=14051]: Uploading segments... Aug 7 23:19:15.896501153 (13h39m50.411603732s) [upload=14051]: Got next segment. Inline: true Aug 7 23:19:15.896505978 (13h39m50.411608555s) [upload=14051]: Waiting for error group managing segments... Aug 7 23:19:15.896570054 (13h39m50.411672631s) [upload=14051]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:19:15.896574888 (13h39m50.411677465s) [upload=14051]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:19:15.896578764 (13h39m50.411681341s) [upload=14051]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:19:15.976040109 (13h39m50.491142676s) [upload=14051]: Done uploading segments... Aug 7 23:19:15.976057062 (13h39m50.491159637s) [upload=14051]: Upload finished. err: <nil> Aug 7 23:19:50.253965481 (13h40m24.76906806s) [upload=14062]: Starting upload Aug 7 23:19:50.254050632 (13h40m24.769153219s) [upload=14062]: Done starting upload Aug 7 23:19:50.254136562 (13h40m24.769239133s) [upload=14062]: Uploading segments... Aug 7 23:19:50.254189310 (13h40m24.769291896s) [upload=14062]: Got next segment. Inline: true Aug 7 23:19:50.254194507 (13h40m24.769297084s) [upload=14062]: Waiting for error group managing segments... Aug 7 23:19:50.254255974 (13h40m24.769358555s) [upload=14062]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:19:50.254262645 (13h40m24.769365222s) [upload=14062]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:19:50.254267031 (13h40m24.769369607s) [upload=14062]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:20:50.224097554 (13h41m24.739200131s) [upload=14062]: Done uploading segments... Aug 7 23:20:50.224116508 (13h41m24.739219086s) [upload=14062]: Upload finished. err: metaclient: context deadline exceeded Duration: 209.454074ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 9 out of 60 is failed (threshold=3) [......X..XXXXX.X.XX.........................................] duration: 225.334617ms FAIL (2025-08-07 23:52:04 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 225.362692ms core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [......X.XXX.XX..............................................] duration: 679.747548ms FAIL (2025-08-07 23:52:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 679.779312ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........XXX..X.X.X.....................................X.] duration: 545.524881ms FAIL (2025-08-07 23:52:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 545.543547ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........XXXXX.............................................] duration: 1.778593407s FAIL (2025-08-07 23:51:59 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.778621171s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 29.042489ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 23:51:55 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 29.070599ms edge gateway vhost bgp us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 31.266467ms target: https://us1.storj.io/login FAIL (2025-08-07 23:51:49 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 31.286693ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 4 out of 60 is failed (threshold=3) [......XX..X...X.............................................] duration: 220.805031ms FAIL (2025-08-07 23:51:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 162725354 (13h31m20.67782793s) [upload=13919]: Starting upload Aug 7 23:10:46.162807728 (13h31m20.677910304s) [upload=13919]: Done starting upload Aug 7 23:10:46.162877771 (13h31m20.677980347s) [upload=13919]: Uploading segments... Aug 7 23:10:46.162939319 (13h31m20.678041901s) [upload=13919]: Got next segment. Inline: true Aug 7 23:10:46.162944044 (13h31m20.678046624s) [upload=13919]: Waiting for error group managing segments... Aug 7 23:10:46.162986704 (13h31m20.678089286s) [upload=13919]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:46.162991049 (13h31m20.678093629s) [upload=13919]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:46.162994213 (13h31m20.678096792s) [upload=13919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:49.331481110 (13h31m23.84658369s) [upload=13919]: Done uploading segments... Aug 7 23:10:49.331502710 (13h31m23.846605285s) [upload=13919]: Upload finished. err: <nil> Aug 7 23:11:56.588799333 (13h32m31.103901909s) [upload=13942]: Starting upload Aug 7 23:11:56.588871172 (13h32m31.103973747s) [upload=13942]: Done starting upload Aug 7 23:11:56.588959193 (13h32m31.104061755s) [upload=13942]: Uploading segments... Aug 7 23:11:56.588996486 (13h32m31.104099072s) [upload=13942]: Got next segment. Inline: true Aug 7 23:11:56.589000914 (13h32m31.104103491s) [upload=13942]: Waiting for error group managing segments... Aug 7 23:11:56.589047870 (13h32m31.104150433s) [upload=13942]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:56.589051856 (13h32m31.104154432s) [upload=13942]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:56.589054372 (13h32m31.10415695s) [upload=13942]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:56.555383755 (13h33m31.070486344s) [upload=13942]: Done uploading segments... Aug 7 23:12:56.555407795 (13h33m31.070510378s) [upload=13942]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:12:56.669023639 (13h33m31.184126201s) [upload=13959]: Starting upload Aug 7 23:12:56.669081512 (13h33m31.184184088s) [upload=13959]: Done starting upload Aug 7 23:12:56.669139265 (13h33m31.184241827s) [upload=13959]: Uploading segments... Aug 7 23:12:56.669181725 (13h33m31.184284307s) [upload=13959]: Got next segment. Inline: true Aug 7 23:12:56.669185727 (13h33m31.18428829s) [upload=13959]: Waiting for error group managing segments... Aug 7 23:12:56.669222629 (13h33m31.184325191s) [upload=13959]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:56.669226085 (13h33m31.18432866s) [upload=13959]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:56.669228556 (13h33m31.184331118s) [upload=13959]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.639927224 (13h34m31.155029791s) [upload=13959]: Done uploading segments... Aug 7 23:13:56.639946863 (13h34m31.155049429s) [upload=13959]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:56.754611797 (13h34m31.269714367s) [upload=13974]: Starting upload Aug 7 23:13:56.754690544 (13h34m31.269793123s) [upload=13974]: Done starting upload Aug 7 23:13:56.754787910 (13h34m31.269890478s) [upload=13974]: Uploading segments... Aug 7 23:13:56.754829279 (13h34m31.269931864s) [upload=13974]: Got next segment. Inline: true Aug 7 23:13:56.754833834 (13h34m31.269936411s) [upload=13974]: Waiting for error group managing segments... Aug 7 23:13:56.754887650 (13h34m31.269990217s) [upload=13974]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:56.754891993 (13h34m31.26999456s) [upload=13974]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:56.754895495 (13h34m31.269998071s) [upload=13974]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:56.814228870 (13h34m31.32933145s) [upload=13974]: Done uploading segments... Aug 7 23:13:56.814253732 (13h34m31.329356295s) [upload=13974]: Upload finished. err: <nil> Aug 7 23:14:51.237083244 (13h35m25.75218581s) [upload=13984]: Starting upload Aug 7 23:14:51.237154440 (13h35m25.752257016s) [upload=13984]: Done starting upload Aug 7 23:14:51.237222602 (13h35m25.752325168s) [upload=13984]: Uploading segments... Aug 7 23:14:51.237272573 (13h35m25.752375158s) [upload=13984]: Got next segment. Inline: true Aug 7 23:14:51.237277088 (13h35m25.752379654s) [upload=13984]: Waiting for error group managing segments... Aug 7 23:14:51.237328372 (13h35m25.752430937s) [upload=13984]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:51.237332974 (13h35m25.75243555s) [upload=13984]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:51.237336117 (13h35m25.752438684s) [upload=13984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:14:51.367347705 (13h35m25.882450281s) [upload=13984]: Done uploading segments... Aug 7 23:14:51.367368302 (13h35m25.882470865s) [upload=13984]: Upload finished. err: <nil> Aug 7 23:16:05.237199068 (13h36m39.752301655s) [upload=14002]: Starting upload Aug 7 23:16:05.237269888 (13h36m39.752372457s) [upload=14002]: Done starting upload Aug 7 23:16:05.237360382 (13h36m39.752462952s) [upload=14002]: Uploading segments... Aug 7 23:16:05.237841750 (13h36m39.752944336s) [upload=14002]: Got next segment. Inline: true Aug 7 23:16:05.237859632 (13h36m39.752962218s) [upload=14002]: Waiting for error group managing segments... Aug 7 23:16:05.237930004 (13h36m39.753032586s) [upload=14002]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:05.237936533 (13h36m39.753039109s) [upload=14002]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:05.237940573 (13h36m39.753043148s) [upload=14002]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:05.205359206 (13h37m39.720461782s) [upload=14002]: Done uploading segments... Aug 7 23:17:05.205376470 (13h37m39.720479046s) [upload=14002]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:17:15.921193047 (13h37m50.436295623s) [upload=14020]: Starting upload Aug 7 23:17:15.921258731 (13h37m50.436361294s) [upload=14020]: Done starting upload Aug 7 23:17:15.921299906 (13h37m50.436402469s) [upload=14020]: Uploading segments... Aug 7 23:17:15.921331213 (13h37m50.436433799s) [upload=14020]: Got next segment. Inline: true Aug 7 23:17:15.921334685 (13h37m50.436437248s) [upload=14020]: Waiting for error group managing segments... Aug 7 23:17:15.921366597 (13h37m50.436469172s) [upload=14020]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:17:15.921370316 (13h37m50.436472878s) [upload=14020]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:17:15.921372605 (13h37m50.436475167s) [upload=14020]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:15.996901858 (13h37m50.512004437s) [upload=14020]: Done uploading segments... Aug 7 23:17:15.996920869 (13h37m50.512023431s) [upload=14020]: Upload finished. err: <nil> Aug 7 23:18:10.846312036 (13h38m45.361414612s) [upload=14034]: Starting upload Aug 7 23:18:10.846377244 (13h38m45.36147981s) [upload=14034]: Done starting upload Aug 7 23:18:10.846461339 (13h38m45.361563904s) [upload=14034]: Uploading segments... Aug 7 23:18:10.846496473 (13h38m45.361599058s) [upload=14034]: Got next segment. Inline: true Aug 7 23:18:10.846501164 (13h38m45.36160374s) [upload=14034]: Waiting for error group managing segments... Aug 7 23:18:10.846554786 (13h38m45.361657362s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:18:10.846559294 (13h38m45.361661871s) [upload=14034]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:18:10.846562922 (13h38m45.361665498s) [upload=14034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:11.200018957 (13h38m45.715121534s) [upload=14034]: Done uploading segments... Aug 7 23:18:11.200049582 (13h38m45.715152158s) [upload=14034]: Upload finished. err: <nil> Aug 7 23:19:15.896167300 (13h39m50.411269878s) [upload=14051]: Starting upload Aug 7 23:19:15.896353995 (13h39m50.411456637s) [upload=14051]: Done starting upload Aug 7 23:19:15.896436268 (13h39m50.411538832s) [upload=14051]: Uploading segments... Aug 7 23:19:15.896501153 (13h39m50.411603732s) [upload=14051]: Got next segment. Inline: true Aug 7 23:19:15.896505978 (13h39m50.411608555s) [upload=14051]: Waiting for error group managing segments... Aug 7 23:19:15.896570054 (13h39m50.411672631s) [upload=14051]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:19:15.896574888 (13h39m50.411677465s) [upload=14051]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:19:15.896578764 (13h39m50.411681341s) [upload=14051]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:19:15.976040109 (13h39m50.491142676s) [upload=14051]: Done uploading segments... Aug 7 23:19:15.976057062 (13h39m50.491159637s) [upload=14051]: Upload finished. err: <nil> Aug 7 23:19:50.253965481 (13h40m24.76906806s) [upload=14062]: Starting upload Aug 7 23:19:50.254050632 (13h40m24.769153219s) [upload=14062]: Done starting upload Aug 7 23:19:50.254136562 (13h40m24.769239133s) [upload=14062]: Uploading segments... Aug 7 23:19:50.254189310 (13h40m24.769291896s) [upload=14062]: Got next segment. Inline: true Aug 7 23:19:50.254194507 (13h40m24.769297084s) [upload=14062]: Waiting for error group managing segments... Aug 7 23:19:50.254255974 (13h40m24.769358555s) [upload=14062]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:19:50.254262645 (13h40m24.769365222s) [upload=14062]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:19:50.254267031 (13h40m24.769369607s) [upload=14062]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:20:50.224097554 (13h41m24.739200131s) [upload=14062]: Done uploading segments... Aug 7 23:20:50.224116508 (13h41m24.739219086s) [upload=14062]: Upload finished. err: metaclient: context deadline exceeded Duration: 220.833452ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 9 out of 60 is failed (threshold=3) [.......X..XXXXX.X.XX........................................] duration: 243.659268ms FAIL (2025-08-07 23:51:48 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 243.683043ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........XX.XXXX...X.X......................................] duration: 1.349933015s FAIL (2025-08-07 23:51:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== ce_id=GH7IOGMAIMB7DEPPOCFSWD2JIW2T3GGESTVFX4YHDDODF2UKVV4A, seg_pos={0 0}, upload=14080]: Uploading piece... Aug 7 23:20:50.270963124 (13h41m24.786065727s) [noise=true , address=104.36.82.61:4398 , piece_id=6JGM2OSGIKGQJVSGKTYSKXWFWUPD2CZBJE5SUVHZTHIDXKYWSVFQ, seg_pos={0 0}, upload=14080]: Uploading piece... Aug 7 23:20:50.271303186 (13h41m24.786405766s) [noise=true , address=45.91.116.14:4198 , piece_id=PPA53436A4JFKNVU3PBE7WSTEWHVNITSNMU2VYCZU5DDZCFJOPSQ, seg_pos={0 0}, upload=14080]: Uploading piece... Aug 7 23:20:50.271556719 (13h41m24.786659309s) [noise=true , address=104.36.82.9:4034 , piece_id=UDZJLVKKAFC3IJGML35BAWKAYPYKCAGODNJBAK4R6QKEQDS5EZSA, seg_pos={0 0}, upload=14080]: Uploading piece... Aug 7 23:20:50.271787526 (13h41m24.786890105s) [noise=true , address=104.36.82.42:4808 , piece_id=G6LVWECALGP45ILRGEK3X3UHOOQHP2UREVWFCDPYJZC3DFMPNYMA, seg_pos={0 0}, upload=14080]: Uploading piece... Aug 7 23:20:50.272659277 (13h41m24.787761864s) [noise=true , address=64.49.13.27:4967 , piece_id=53RPNRVKZ6TYT7VP2DCXAX33YMUMDAWZS3RMRS53AW3OSEDIPCTA, seg_pos={0 0}, upload=14080]: Uploading piece... Aug 7 23:20:50.272784283 (13h41m24.787886875s) [noise=true , address=104.36.82.16:4326 , piece_id=DQUUJR7LJFQJQUKRTWXHS5SCMU4ZZ72EIYZZJY25HRM2BJYHLR7A, seg_pos={0 0}, upload=14080]: Uploading piece... Aug 7 23:20:50.273228084 (13h41m24.788330665s) [noise=true , address=104.36.82.60:4353 , piece_id=T26XKES2P2UK4QHLVWKJUZNS2IXZUYWC7FU45LGU7SC2SJILXFMA, seg_pos={0 0}, upload=14080]: Uploading piece... Aug 7 23:20:50.273443857 (13h41m24.788546457s) [noise=true , address=104.36.82.44:4900 , piece_id=PS3WKIHAI4T25TOTOGXJH5RMXCCKYJQJ5JAX52ODW2G24VXLZUSQ, seg_pos={0 0}, upload=14080]: Uploading piece... Aug 7 23:20:50.274091759 (13h41m24.789194346s) [noise=true , address=104.36.82.2:4759 , piece_id=WQ42DKSHG5FXUCDKII7BNEFDA25UX6Q3ARG3DXTHV7OKIBM5NHEQ, seg_pos={0 0}, upload=14080]: Uploading piece... Aug 7 23:20:50.275262302 (13h41m24.790364888s) [noise=true , address=79.127.220.97:4396 , piece_id=UQGXEYE4PKBTNQO2W73CX3W5IKMVDVDDSN3ZXYOWIJGTREKQF5VQ, seg_pos={0 0}, upload=14080]: Uploading piece... Aug 7 23:20:50.275755725 (13h41m24.790858314s) [noise=true , address=104.36.82.56:4236 , piece_id=PTBEE4LERZ3JO3OPPJFAIBQ5LYSTEDZCVU2FSA7QZFZJUK7Q5TQA, seg_pos={0 0}, upload=14080]: Uploading piece... Aug 7 23:20:50.276176266 (13h41m24.791278854s) [noise=true , address=104.36.82.4:4811 , piece_id=T3ZIA6J7HELFDWFFQUTJK7LGIMXOKOV5ZGURFEIKGJRBU4YWW3TA, seg_pos={0 0}, upload=14080]: Uploading piece... Aug 7 23:20:50.276690914 (13h41m24.791793502s) [noise=true , address=45.91.116.10:4104 , piece_id=ZIUMAHVVTT6X7ZVWOJI2ELFYQG7QXFP2W4CTX2PWDSCSBIZU2TMA, seg_pos={0 0}, upload=14080]: Uploading piece... Aug 7 23:20:50.277946066 (13h41m24.793048658s) [noise=true , address=156.146.43.225:4447 , piece_id=QMDN5UIK3XYXB2X35EVI33U2ANOEWIU5XUU2A7CFYRSRRJLDFW4Q, seg_pos={0 0}, upload=14080]: Uploading piece... Aug 7 23:20:50.278411190 (13h41m24.793513776s) [noise=true , address=104.36.82.23:4463 , piece_id=AF7VUSU4PDWMINEEJ3CLAQ5GYANWSBTB7ZX3KK7BBPWKX575DY5A, seg_pos={0 0}, upload=14080]: Uploading piece... Aug 7 23:20:50.363417372 (13h41m24.878519955s) [upload=14080]: Next returned nil segment Aug 7 23:20:50.363434233 (13h41m24.878536817s) [upload=14080]: Waiting for error group managing segments... Aug 7 23:20:50.370437337 (13h41m24.885539907s) [noise=true , address=45.91.116.14:4198 , piece_id=PPA53436A4JFKNVU3PBE7WSTEWHVNITSNMU2VYCZU5DDZCFJOPSQ, seg_pos={0 0}, upload=14080]: Done uploading piece. err: <nil> Aug 7 23:20:50.371668429 (13h41m24.886771s) [noise=true , address=104.36.82.56:4236 , piece_id=PTBEE4LERZ3JO3OPPJFAIBQ5LYSTEDZCVU2FSA7QZFZJUK7Q5TQA, seg_pos={0 0}, upload=14080]: Done uploading piece. err: <nil> Aug 7 23:20:50.375301858 (13h41m24.890404435s) [noise=true , address=104.36.82.23:4463 , piece_id=AF7VUSU4PDWMINEEJ3CLAQ5GYANWSBTB7ZX3KK7BBPWKX575DY5A, seg_pos={0 0}, upload=14080]: Done uploading piece. err: <nil> Aug 7 23:20:50.375604014 (13h41m24.890706595s) [noise=true , address=104.36.82.42:4808 , piece_id=G6LVWECALGP45ILRGEK3X3UHOOQHP2UREVWFCDPYJZC3DFMPNYMA, seg_pos={0 0}, upload=14080]: Done uploading piece. err: <nil> Aug 7 23:20:50.375817725 (13h41m24.89092029s) [noise=true , address=104.36.82.47:4535 , piece_id=XUCXMLKMK2JHA5RRZPMYTGC5Z6JB4BFI3J6BT5RM65O4KEEI3LEA, seg_pos={0 0}, upload=14080]: Done uploading piece. err: <nil> Aug 7 23:20:50.376297840 (13h41m24.89140042s) [noise=true , address=104.36.82.4:4811 , piece_id=T3ZIA6J7HELFDWFFQUTJK7LGIMXOKOV5ZGURFEIKGJRBU4YWW3TA, seg_pos={0 0}, upload=14080]: Done uploading piece. err: <nil> Aug 7 23:20:50.376376567 (13h41m24.89147913s) [noise=true , address=104.36.82.9:4034 , piece_id=UDZJLVKKAFC3IJGML35BAWKAYPYKCAGODNJBAK4R6QKEQDS5EZSA, seg_pos={0 0}, upload=14080]: Done uploading piece. err: <nil> Aug 7 23:20:50.378451878 (13h41m24.893554445s) [noise=true , address=104.36.82.37:4673 , piece_id=GH7IOGMAIMB7DEPPOCFSWD2JIW2T3GGESTVFX4YHDDODF2UKVV4A, seg_pos={0 0}, upload=14080]: Done uploading piece. err: <nil> Aug 7 23:20:50.387814655 (13h41m24.902917224s) [noise=true , address=104.36.82.2:4759 , piece_id=WQ42DKSHG5FXUCDKII7BNEFDA25UX6Q3ARG3DXTHV7OKIBM5NHEQ, seg_pos={0 0}, upload=14080]: Done uploading piece. err: <nil> Aug 7 23:20:50.393629027 (13h41m24.908731604s) [noise=true , address=45.91.116.10:4104 , piece_id=ZIUMAHVVTT6X7ZVWOJI2ELFYQG7QXFP2W4CTX2PWDSCSBIZU2TMA, seg_pos={0 0}, upload=14080]: Done uploading piece. err: <nil> Aug 7 23:20:50.399762700 (13h41m24.914865269s) [noise=true , address=104.36.82.60:4353 , piece_id=T26XKES2P2UK4QHLVWKJUZNS2IXZUYWC7FU45LGU7SC2SJILXFMA, seg_pos={0 0}, upload=14080]: Done uploading piece. err: <nil> Aug 7 23:20:50.461291683 (13h41m24.976394246s) [noise=true , address=79.127.220.97:4396 , piece_id=UQGXEYE4PKBTNQO2W73CX3W5IKMVDVDDSN3ZXYOWIJGTREKQF5VQ, seg_pos={0 0}, upload=14080]: Done uploading piece. err: <nil> Aug 7 23:20:50.478584042 (13h41m24.993686633s) [noise=true , address=104.36.82.61:4398 , piece_id=6JGM2OSGIKGQJVSGKTYSKXWFWUPD2CZBJE5SUVHZTHIDXKYWSVFQ, seg_pos={0 0}, upload=14080]: Done uploading piece. err: <nil> Aug 7 23:20:50.520830872 (13h41m25.035933449s) [noise=true , address=156.146.43.225:4447 , piece_id=QMDN5UIK3XYXB2X35EVI33U2ANOEWIU5XUU2A7CFYRSRRJLDFW4Q, seg_pos={0 0}, upload=14080]: Done uploading piece. err: <nil> Aug 7 23:20:50.520859322 (13h41m25.035961898s) [seg_pos={0 0}, upload=14080]: Segment reached optimal threshold of 14 pieces. Aug 7 23:20:50.521038126 (13h41m25.036140694s) [noise=true , address=104.36.82.44:4900 , piece_id=PS3WKIHAI4T25TOTOGXJH5RMXCCKYJQJ5JAX52ODW2G24VXLZUSQ, seg_pos={0 0}, upload=14080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1R8K65zsCspu7W2oAnvtpz7zEsQFkDG1e5BAp6faQYMQrkh4oQ): protocol: expected piece hash; context canceled; EOF Aug 7 23:20:50.521126310 (13h41m25.03622888s) [noise=true , address=104.36.82.5:4842 , piece_id=AOQNSYWK7J4PGSTM5JEABQ34AFL33ZNKJRI77WO6GY5RKO7DAIHQ, seg_pos={0 0}, upload=14080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn): protocol: write tcp 159.223.143.39:48910->104.36.82.5:4842: use of closed network connection; write tcp 159.223.143.39:48910->104.36.82.5:4842: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48910->104.36.82.5:4842: use of closed network connection Aug 7 23:20:50.521313827 (13h41m25.036416407s) [noise=true , address=104.36.82.16:4326 , piece_id=DQUUJR7LJFQJQUKRTWXHS5SCMU4ZZ72EIYZZJY25HRM2BJYHLR7A, seg_pos={0 0}, upload=14080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pU8MZZKS88CPFdZoaQ4msNfkCbCewTmS2wKEdN7pG5aCWoAr6): protocol: read tcp 159.223.143.39:35084->104.36.82.16:4326: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:20:50.521375844 (13h41m25.03647841s) [noise=true , address=64.49.13.27:4967 , piece_id=53RPNRVKZ6TYT7VP2DCXAX33YMUMDAWZS3RMRS53AW3OSEDIPCTA, seg_pos={0 0}, upload=14080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nVt2Ze4c5jQTyqFQDsJVaZKqge6gfjwpxUZXDCx8UVcySk4r8): protocol: read tcp 159.223.143.39:54110->64.49.13.27:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 23:20:50.521411430 (13h41m25.036513997s) [seg_pos={0 0}, upload=14080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 23:20:50.521520073 (13h41m25.03662264s) [upload=14080]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:20:50.521525409 (13h41m25.036627984s) [upload=14080]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:20:50.573889476 (13h41m25.088992052s) [upload=14080]: Done uploading segments... Aug 7 23:20:50.573905889 (13h41m25.089008466s) [upload=14080]: Upload finished. err: <nil> Aug 7 23:21:30.905760263 (13h42m5.42086284s) [upload=14094]: Starting upload Aug 7 23:21:30.905840100 (13h42m5.420942676s) [upload=14094]: Done starting upload Aug 7 23:21:30.905958959 (13h42m5.421061527s) [upload=14094]: Uploading segments... Aug 7 23:21:30.906533404 (13h42m5.421636042s) [upload=14094]: Got next segment. Inline: false Aug 7 23:21:30.906593461 (13h42m5.421696028s) [upload=14094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:21:30.906598444 (13h42m5.42170102s) [upload=14094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:22:30.875535565 (13h43m5.390638142s) [upload=14094]: Next segment err: context deadline exceeded Aug 7 23:22:30.875554779 (13h43m5.390657361s) [upload=14094]: Done uploading segments... Aug 7 23:22:30.875570371 (13h43m5.390672939s) [upload=14094]: Upload finished. err: context deadline exceeded Duration: 1.349958095s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 598.920724ms FAIL (2025-08-07 23:51:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:11:04.814410393 (13h31m39.329512969s) [upload=13925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:11:04.814415016 (13h31m39.329517592s) [upload=13925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:04.947583472 (13h31m39.462686071s) [upload=13925]: Done uploading segments... Aug 7 23:11:04.947609736 (13h31m39.462712324s) [upload=13925]: Upload finished. err: <nil> Aug 7 23:11:29.227536921 (13h32m3.742639487s) [upload=13931]: Starting upload Aug 7 23:11:29.227600047 (13h32m3.742702623s) [upload=13931]: Done starting upload Aug 7 23:11:29.227715021 (13h32m3.742817591s) [upload=13931]: Uploading segments... Aug 7 23:11:29.227850209 (13h32m3.74295278s) [upload=13931]: Got next segment. Inline: false Aug 7 23:11:29.227898592 (13h32m3.743001169s) [upload=13931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:29.227903365 (13h32m3.743005942s) [upload=13931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:11:47.514990540 (13h32m22.03009312s) [seg_pos={0 0}, upload=13931]: Begin upload segment... Aug 7 23:11:47.515067094 (13h32m22.030169657s) [seg_pos={0 0}, upload=13931]: Done begin upload segment. Aug 7 23:11:47.515129228 (13h32m22.030231814s) [noise=true , address=77.90.31.18:4806 , piece_id=LI5HUM3HCYFYYSO6BENU4DQ2SFOPEJLRWZRM5PWCFXUZON4WCMAA, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.515554241 (13h32m22.03065682s) [noise=true , address=104.36.82.14:4222 , piece_id=B5WB353FQ47THJP7MXV4AY5Z6FFSG7A66X72BLJUDK2ZTM3FNDVA, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.516068554 (13h32m22.031171143s) [noise=true , address=45.91.116.7:4010 , piece_id=XOROULN3EEUSG45LZ5V53ORP2YV7CWLRQDSQ2AXREAFB62FGNL6A, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.516625763 (13h32m22.031728346s) [noise=true , address=45.91.116.8:4047 , piece_id=5E2VVSY65OAD6OFCQBBOTYLCLUKNVCVD3KY7OOVBI5TCYKQUKBQQ, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.517060179 (13h32m22.03216276s) [noise=true , address=104.36.82.7:4910 , piece_id=Y75KUQR2IYRZBANBQPDFZTUJZPM54BA3KI6ZEF77NI3V755HPBOA, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.517396640 (13h32m22.032499217s) [noise=true , address=104.36.82.6:4873 , piece_id=3YGEWHQ7MXU6IPUJCYNM7L4DMNOECDSTVXMUBCX3CJCGPJ4OGGIQ, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.517801490 (13h32m22.032904072s) [noise=true , address=45.91.116.10:4081 , piece_id=WDJ6KCU4LP75N7TSSR2VTWWEON3VRWE7BFJL57H2RM7ZYI2GYLWQ, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.518135215 (13h32m22.033237797s) [noise=true , address=45.91.116.5:4297 , piece_id=T76WY44JQ55TJH4KJM57QWNBP7DVYNIBDVAPVN2AUTWASN4V76IA, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.518441967 (13h32m22.033544543s) [noise=true , address=77.90.31.11:4553 , piece_id=NK6AJDEDIAA5N3DNKBIPKZXURRSBOJG26FIMUB2SVFTIWOKFJWNQ, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.518756911 (13h32m22.033859486s) [noise=true , address=77.90.31.21:4877 , piece_id=NVD7UBISRZJFOBJR7JKOX6V354Z6VCEP5TWJPGW3WOSI6S6K5FDA, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.519217404 (13h32m22.034319986s) [noise=true , address=45.91.116.6:4973 , piece_id=HEA5GQ53KZVIMBZ5LC3S4Y4372WMGUSATCBDTY5BIHZ4WL2PG7HQ, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.519516685 (13h32m22.034619271s) [noise=true , address=77.90.31.19:4841 , piece_id=Z76CMQDB3DDLHQV35DZORZWRBX2D52BLBDCKPFXPONE3OUNGSGHA, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.519904148 (13h32m22.035006727s) [noise=true , address=77.90.31.19:4842 , piece_id=Q7PUVWG4BXEJSS4YOBVTMUHCAWWKKQAR43NH4U3R24YMK3MP6H6Q, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.520220651 (13h32m22.035323239s) [noise=true , address=77.90.31.10:4517 , piece_id=WJ7TMWHMURKZ6HJON2NBIZVIC4XEXUUR5M7KD5PFWV4MEB3DUV7A, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.520644038 (13h32m22.035746619s) [noise=true , address=104.36.82.12:4149 , piece_id=UEFYG73QPD3OQX5IF2BGE7SQHYL5OKV37LQOKGNECBGRYQLULCJQ, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.521219880 (13h32m22.036322466s) [noise=true , address=77.90.31.17:4771 , piece_id=YCZ26U3QI4NQ7CKNVZEEB4O6XK4ZR3SPXTKT75P5U4ZQLUWOJWQA, seg_pos={0 0}, upload=13931]: Uploading piece... Aug 7 23:11:47.609483246 (13h32m22.124585826s) [upload=13931]: Next returned nil segment Aug 7 23:11:47.609501768 (13h32m22.124604348s) [upload=13931]: Waiting for error group managing segments... Aug 7 23:11:47.614240475 (13h32m22.129343046s) [noise=true , address=45.91.116.8:4047 , piece_id=5E2VVSY65OAD6OFCQBBOTYLCLUKNVCVD3KY7OOVBI5TCYKQUKBQQ, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.614925369 (13h32m22.130027939s) [noise=true , address=45.91.116.5:4297 , piece_id=T76WY44JQ55TJH4KJM57QWNBP7DVYNIBDVAPVN2AUTWASN4V76IA, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.615059457 (13h32m22.130162026s) [noise=true , address=45.91.116.7:4010 , piece_id=XOROULN3EEUSG45LZ5V53ORP2YV7CWLRQDSQ2AXREAFB62FGNL6A, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.615458775 (13h32m22.130561338s) [noise=true , address=77.90.31.19:4842 , piece_id=Q7PUVWG4BXEJSS4YOBVTMUHCAWWKKQAR43NH4U3R24YMK3MP6H6Q, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.616116276 (13h32m22.131218838s) [noise=true , address=77.90.31.18:4806 , piece_id=LI5HUM3HCYFYYSO6BENU4DQ2SFOPEJLRWZRM5PWCFXUZON4WCMAA, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.616661348 (13h32m22.131763909s) [noise=true , address=77.90.31.17:4771 , piece_id=YCZ26U3QI4NQ7CKNVZEEB4O6XK4ZR3SPXTKT75P5U4ZQLUWOJWQA, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.618000416 (13h32m22.133102978s) [noise=true , address=45.91.116.10:4081 , piece_id=WDJ6KCU4LP75N7TSSR2VTWWEON3VRWE7BFJL57H2RM7ZYI2GYLWQ, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.618530300 (13h32m22.133632866s) [noise=true , address=45.91.116.6:4973 , piece_id=HEA5GQ53KZVIMBZ5LC3S4Y4372WMGUSATCBDTY5BIHZ4WL2PG7HQ, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.619098772 (13h32m22.13420134s) [noise=true , address=77.90.31.19:4841 , piece_id=Z76CMQDB3DDLHQV35DZORZWRBX2D52BLBDCKPFXPONE3OUNGSGHA, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.619697172 (13h32m22.13479975s) [noise=true , address=77.90.31.11:4553 , piece_id=NK6AJDEDIAA5N3DNKBIPKZXURRSBOJG26FIMUB2SVFTIWOKFJWNQ, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.620954558 (13h32m22.136057128s) [noise=true , address=77.90.31.21:4877 , piece_id=NVD7UBISRZJFOBJR7JKOX6V354Z6VCEP5TWJPGW3WOSI6S6K5FDA, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.627010966 (13h32m22.142113543s) [noise=true , address=104.36.82.7:4910 , piece_id=Y75KUQR2IYRZBANBQPDFZTUJZPM54BA3KI6ZEF77NI3V755HPBOA, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.629783113 (13h32m22.144885677s) [noise=true , address=77.90.31.10:4517 , piece_id=WJ7TMWHMURKZ6HJON2NBIZVIC4XEXUUR5M7KD5PFWV4MEB3DUV7A, seg_pos={0 0}, upload=13931]: Done uploading piece. err: <nil> Aug 7 23:11:47.629816031 (13h32m22.144918609s) [seg_pos={0 0}, upload=13931]: Segment reached optimal threshold of 13 pieces. Aug 7 23:11:47.630126623 (13h32m22.145229201s) [noise=true , address=104.36.82.14:4222 , piece_id=B5WB353FQ47THJP7MXV4AY5Z6FFSG7A66X72BLJUDK2ZTM3FNDVA, seg_pos={0 0}, upload=13931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 159.223.143.39:47404->104.36.82.14:4222: use of closed network connection; write tcp 159.223.143.39:47404->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:47404->104.36.82.14:4222: use of closed network connection Aug 7 23:11:47.630137290 (13h32m22.145239874s) [noise=true , address=104.36.82.12:4149 , piece_id=UEFYG73QPD3OQX5IF2BGE7SQHYL5OKV37LQOKGNECBGRYQLULCJQ, seg_pos={0 0}, upload=13931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm): protocol: write tcp 159.223.143.39:35622->104.36.82.12:4149: use of closed network connection; write tcp 159.223.143.39:35622->104.36.82.12:4149: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:35622->104.36.82.12:4149: use of closed network connection Aug 7 23:11:47.630232787 (13h32m22.145335361s) [noise=true , address=104.36.82.6:4873 , piece_id=3YGEWHQ7MXU6IPUJCYNM7L4DMNOECDSTVXMUBCX3CJCGPJ4OGGIQ, seg_pos={0 0}, upload=13931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: read tcp 159.223.143.39:36062->104.36.82.6:4873: use of closed network connection; context canceled Aug 7 23:11:47.630264689 (13h32m22.145367273s) [seg_pos={0 0}, upload=13931]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:11:47.630345104 (13h32m22.145447682s) [upload=13931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:11:47.630350066 (13h32m22.145452652s) [upload=13931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:18:37.334152859 (13h39m11.849255428s) [upload=13931]: Done uploading segments... Aug 7 23:18:37.334173589 (13h39m11.849276166s) [upload=13931]: Upload finished. err: metaclient: context deadline exceeded Duration: 598.95113ms core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 40.752311ms target: https://us1.storj.io/login FAIL (2025-08-07 23:51:33 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 40.778349ms core satellite us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 36.161125ms remote: "tcp:136.0.77.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 23:51:24 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 36.181837ms edge gateway vhost bgp us1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 61.706782ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 23:51:19 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 61.727571ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [.......XX.X.X....XX.........................................] duration: 348.430187ms FAIL (2025-08-07 23:51:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 348.453841ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 5 out of 60 is failed (threshold=3) [.........XX.XXX.............................................] duration: 8.740285858s FAIL (2025-08-07 23:51:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 30m31.135428317s) [upload=13902]: Starting upload Aug 7 23:09:56.620515140 (13h30m31.135617722s) [upload=13902]: Done starting upload Aug 7 23:09:56.620600541 (13h30m31.135703106s) [upload=13902]: Uploading segments... Aug 7 23:09:56.620663979 (13h30m31.135766559s) [upload=13902]: Got next segment. Inline: true Aug 7 23:09:56.620669313 (13h30m31.13577189s) [upload=13902]: Waiting for error group managing segments... Aug 7 23:09:56.620725144 (13h30m31.13582772s) [upload=13902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:09:56.620729700 (13h30m31.135832276s) [upload=13902]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:09:56.620732858 (13h30m31.135835435s) [upload=13902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:09:56.756931332 (13h30m31.272033912s) [upload=13902]: Done uploading segments... Aug 7 23:09:56.756952203 (13h30m31.272054766s) [upload=13902]: Upload finished. err: <nil> Aug 7 23:10:22.255312070 (13h30m56.770414636s) [upload=13913]: Starting upload Aug 7 23:10:22.255389033 (13h30m56.770491609s) [upload=13913]: Done starting upload Aug 7 23:10:22.255538785 (13h30m56.77064136s) [upload=13913]: Uploading segments... Aug 7 23:10:22.255632108 (13h30m56.77073469s) [upload=13913]: Got next segment. Inline: true Aug 7 23:10:22.255639107 (13h30m56.770741683s) [upload=13913]: Waiting for error group managing segments... Aug 7 23:10:22.255686584 (13h30m56.77078915s) [upload=13913]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:10:22.255691754 (13h30m56.770794331s) [upload=13913]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:10:22.255695470 (13h30m56.770798046s) [upload=13913]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:10:22.322045125 (13h30m56.837147702s) [upload=13913]: Done uploading segments... Aug 7 23:10:22.322068753 (13h30m56.837171329s) [upload=13913]: Upload finished. err: <nil> Aug 7 23:11:00.342239204 (13h31m34.85734177s) [upload=13922]: Starting upload Aug 7 23:11:00.342310667 (13h31m34.857413235s) [upload=13922]: Done starting upload Aug 7 23:11:00.342383830 (13h31m34.85748641s) [upload=13922]: Uploading segments... Aug 7 23:11:00.342449267 (13h31m34.857551854s) [upload=13922]: Got next segment. Inline: true Aug 7 23:11:00.342455504 (13h31m34.857558092s) [upload=13922]: Waiting for error group managing segments... Aug 7 23:11:00.342503627 (13h31m34.857606209s) [upload=13922]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:00.342508465 (13h31m34.857611046s) [upload=13922]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:00.342511859 (13h31m34.857614451s) [upload=13922]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:00.401913127 (13h31m34.917015712s) [upload=13922]: Done uploading segments... Aug 7 23:11:00.401931904 (13h31m34.917034472s) [upload=13922]: Upload finished. err: <nil> Aug 7 23:11:35.523175401 (13h32m10.03827798s) [upload=13934]: Starting upload Aug 7 23:11:35.523266566 (13h32m10.038369147s) [upload=13934]: Done starting upload Aug 7 23:11:35.523355068 (13h32m10.038457647s) [upload=13934]: Uploading segments... Aug 7 23:11:35.523405664 (13h32m10.03850825s) [upload=13934]: Got next segment. Inline: true Aug 7 23:11:35.523412393 (13h32m10.038514977s) [upload=13934]: Waiting for error group managing segments... Aug 7 23:11:35.523470368 (13h32m10.038572947s) [upload=13934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:11:35.523479499 (13h32m10.038582083s) [upload=13934]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:11:35.523484240 (13h32m10.038586821s) [upload=13934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:11:35.587656651 (13h32m10.102759216s) [upload=13934]: Done uploading segments... Aug 7 23:11:35.587674771 (13h32m10.102777338s) [upload=13934]: Upload finished. err: <nil> Aug 7 23:12:23.226340957 (13h32m57.741443533s) [upload=13947]: Starting upload Aug 7 23:12:23.226411041 (13h32m57.741513618s) [upload=13947]: Done starting upload Aug 7 23:12:23.226470575 (13h32m57.74157314s) [upload=13947]: Uploading segments... Aug 7 23:12:23.226506745 (13h32m57.741609326s) [upload=13947]: Got next segment. Inline: true Aug 7 23:12:23.226511437 (13h32m57.741614019s) [upload=13947]: Waiting for error group managing segments... Aug 7 23:12:23.226561258 (13h32m57.741663839s) [upload=13947]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:23.226566182 (13h32m57.741668762s) [upload=13947]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:23.226569380 (13h32m57.741671956s) [upload=13947]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:12:53.198277272 (13h33m27.713379848s) [upload=13947]: Done uploading segments... Aug 7 23:12:53.198295252 (13h33m27.713397814s) [upload=13947]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:12:53.310207725 (13h33m27.825310302s) [upload=13957]: Starting upload Aug 7 23:12:53.310290257 (13h33m27.82539282s) [upload=13957]: Done starting upload Aug 7 23:12:53.310379699 (13h33m27.825482276s) [upload=13957]: Uploading segments... Aug 7 23:12:53.310488747 (13h33m27.825591336s) [upload=13957]: Got next segment. Inline: true Aug 7 23:12:53.310497967 (13h33m27.825600548s) [upload=13957]: Waiting for error group managing segments... Aug 7 23:12:53.310564975 (13h33m27.825667562s) [upload=13957]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:12:53.310572430 (13h33m27.825675017s) [upload=13957]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:12:53.310576051 (13h33m27.825678628s) [upload=13957]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:23.280527765 (13h33m57.79563034s) [upload=13957]: Done uploading segments... Aug 7 23:13:23.280561185 (13h33m57.795663762s) [upload=13957]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:13:50.810735096 (13h34m25.325837673s) [upload=13972]: Starting upload Aug 7 23:13:50.810789852 (13h34m25.325892437s) [upload=13972]: Done starting upload Aug 7 23:13:50.810850987 (13h34m25.325953556s) [upload=13972]: Uploading segments... Aug 7 23:13:50.810923216 (13h34m25.326025797s) [upload=13972]: Got next segment. Inline: true Aug 7 23:13:50.810928523 (13h34m25.326031096s) [upload=13972]: Waiting for error group managing segments... Aug 7 23:13:50.810970440 (13h34m25.326073016s) [upload=13972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:13:50.810974710 (13h34m25.326077286s) [upload=13972]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:13:50.810978274 (13h34m25.32608085s) [upload=13972]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:13:50.866577223 (13h34m25.3816798s) [upload=13972]: Done uploading segments... Aug 7 23:13:50.866594654 (13h34m25.38169723s) [upload=13972]: Upload finished. err: <nil> Aug 7 23:14:54.225692687 (13h35m28.740795254s) [upload=13986]: Starting upload Aug 7 23:14:54.225785134 (13h35m28.740887721s) [upload=13986]: Done starting upload Aug 7 23:14:54.225870963 (13h35m28.740973529s) [upload=13986]: Uploading segments... Aug 7 23:14:54.225923809 (13h35m28.741026386s) [upload=13986]: Got next segment. Inline: true Aug 7 23:14:54.225928631 (13h35m28.741031207s) [upload=13986]: Waiting for error group managing segments... Aug 7 23:14:54.225969273 (13h35m28.741071842s) [upload=13986]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:14:54.225973734 (13h35m28.7410763s) [upload=13986]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:14:54.225976991 (13h35m28.741079566s) [upload=13986]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:24.191802298 (13h35m58.706904867s) [upload=13986]: Done uploading segments... Aug 7 23:15:24.191822370 (13h35m58.70692494s) [upload=13986]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:15:24.446619359 (13h35m58.961721924s) [upload=13992]: Starting upload Aug 7 23:15:24.446725754 (13h35m58.961828338s) [upload=13992]: Done starting upload Aug 7 23:15:24.446812887 (13h35m58.961915463s) [upload=13992]: Uploading segments... Aug 7 23:15:24.446871932 (13h35m58.961974519s) [upload=13992]: Got next segment. Inline: true Aug 7 23:15:24.446878746 (13h35m58.961981315s) [upload=13992]: Waiting for error group managing segments... Aug 7 23:15:24.446939653 (13h35m58.962042222s) [upload=13992]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:15:24.446945288 (13h35m58.962047863s) [upload=13992]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:15:24.446950467 (13h35m58.96205303s) [upload=13992]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:15:54.414392139 (13h36m28.929494705s) [upload=13992]: Done uploading segments... Aug 7 23:15:54.414412085 (13h36m28.929514651s) [upload=13992]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:16:32.150263694 (13h37m6.665366261s) [upload=14009]: Starting upload Aug 7 23:16:32.150345034 (13h37m6.665447612s) [upload=14009]: Done starting upload Aug 7 23:16:32.150402507 (13h37m6.665505071s) [upload=14009]: Uploading segments... Aug 7 23:16:32.150453597 (13h37m6.665556176s) [upload=14009]: Got next segment. Inline: true Aug 7 23:16:32.150458385 (13h37m6.665560958s) [upload=14009]: Waiting for error group managing segments... Aug 7 23:16:32.150501065 (13h37m6.665603641s) [upload=14009]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:16:32.150505833 (13h37m6.66560841s) [upload=14009]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:16:32.150509597 (13h37m6.665612174s) [upload=14009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:17:02.044102103 (13h37m36.55920467s) [upload=14009]: Done uploading segments... Aug 7 23:17:02.044123177 (13h37m36.559225743s) [upload=14009]: Upload finished. err: metaclient: context deadline exceeded Duration: 8.740337071s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............XXX..X.X.X.....................................] duration: 539.405268ms FAIL (2025-08-07 23:51:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 539.43365ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 9 out of 60 is failed (threshold=3) [........X..XXXXX.X.XX.......................................] duration: 412.293916ms FAIL (2025-08-07 23:51:05 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 412.315974ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........XXXXX............................................] duration: 1.886629902s FAIL (2025-08-07 23:51:03 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 1.886669182s core uplink us1

Orbiter Check History

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)