Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [....................................X..X....XX...X..........] duration: 572.045803ms FAIL (2025-08-08 00:08:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 572.068885ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 6 out of 60 is failed (threshold=3) [.....................................X.XX.X.....X..X........] duration: 204.008109ms FAIL (2025-08-08 00:08:50 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Upload finished. err: <nil> Aug 7 23:56:22.913855555 (14h16m57.428958117s) [upload=14689]: Starting upload Aug 7 23:56:22.913925154 (14h16m57.429027734s) [upload=14689]: Done starting upload Aug 7 23:56:22.913999531 (14h16m57.429102093s) [upload=14689]: Uploading segments... Aug 7 23:56:22.914048699 (14h16m57.429151282s) [upload=14689]: Got next segment. Inline: true Aug 7 23:56:22.914052623 (14h16m57.4291552s) [upload=14689]: Waiting for error group managing segments... Aug 7 23:56:22.914089928 (14h16m57.429192503s) [upload=14689]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:22.914093434 (14h16m57.42919601s) [upload=14689]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:22.914096175 (14h16m57.429198751s) [upload=14689]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:52.884393262 (14h17m27.399495844s) [upload=14689]: Done uploading segments... Aug 7 23:56:52.884415558 (14h17m27.399518126s) [upload=14689]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:09.822697015 (14h17m44.337799585s) [upload=14701]: Starting upload Aug 7 23:57:09.822774891 (14h17m44.337877456s) [upload=14701]: Done starting upload Aug 7 23:57:09.822876390 (14h17m44.337978955s) [upload=14701]: Uploading segments... Aug 7 23:57:09.823345112 (14h17m44.338447694s) [upload=14701]: Got next segment. Inline: true Aug 7 23:57:09.823357163 (14h17m44.338459742s) [upload=14701]: Waiting for error group managing segments... Aug 7 23:57:09.823423761 (14h17m44.338526328s) [upload=14701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:09.823429578 (14h17m44.338532154s) [upload=14701]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:09.823433695 (14h17m44.338536271s) [upload=14701]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:09.886245350 (14h17m44.401347936s) [upload=14701]: Done uploading segments... Aug 7 23:57:09.886269151 (14h17m44.401371729s) [upload=14701]: Upload finished. err: <nil> Aug 7 23:58:11.646122622 (14h18m46.161225204s) [upload=14719]: Starting upload Aug 7 23:58:11.646203494 (14h18m46.16130607s) [upload=14719]: Done starting upload Aug 7 23:58:11.646278294 (14h18m46.161380873s) [upload=14719]: Uploading segments... Aug 7 23:58:11.646339731 (14h18m46.161442316s) [upload=14719]: Got next segment. Inline: true Aug 7 23:58:11.646344984 (14h18m46.16144756s) [upload=14719]: Waiting for error group managing segments... Aug 7 23:58:11.646388852 (14h18m46.161491429s) [upload=14719]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:11.646393482 (14h18m46.161496048s) [upload=14719]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:11.646396765 (14h18m46.161499342s) [upload=14719]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:11.705007157 (14h18m46.220109734s) [upload=14719]: Done uploading segments... Aug 7 23:58:11.705026176 (14h18m46.220128742s) [upload=14719]: Upload finished. err: <nil> Aug 7 23:58:41.852871585 (14h19m16.367974172s) [upload=14731]: Starting upload Aug 7 23:58:41.852974809 (14h19m16.368077394s) [upload=14731]: Done starting upload Aug 7 23:58:41.853095753 (14h19m16.368198318s) [upload=14731]: Uploading segments... Aug 7 23:58:41.853172118 (14h19m16.368274707s) [upload=14731]: Got next segment. Inline: true Aug 7 23:58:41.853178991 (14h19m16.368281577s) [upload=14731]: Waiting for error group managing segments... Aug 7 23:58:41.853256548 (14h19m16.368359135s) [upload=14731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:41.853263687 (14h19m16.368366273s) [upload=14731]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:41.853268996 (14h19m16.368371576s) [upload=14731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:41.919489485 (14h19m16.434592068s) [upload=14731]: Done uploading segments... Aug 7 23:58:41.919512553 (14h19m16.434615126s) [upload=14731]: Upload finished. err: <nil> Aug 7 23:59:44.072708375 (14h20m18.587810945s) [upload=14749]: Starting upload Aug 7 23:59:44.072763052 (14h20m18.587865629s) [upload=14749]: Done starting upload Aug 7 23:59:44.072818447 (14h20m18.587921015s) [upload=14749]: Uploading segments... Aug 7 23:59:44.072859036 (14h20m18.587961622s) [upload=14749]: Got next segment. Inline: true Aug 7 23:59:44.072863633 (14h20m18.587966209s) [upload=14749]: Waiting for error group managing segments... Aug 7 23:59:44.072902482 (14h20m18.588005058s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:44.072906670 (14h20m18.588009237s) [upload=14749]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:44.072910115 (14h20m18.588012692s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:44.143691065 (14h20m18.658793888s) [upload=14749]: Done uploading segments... Aug 7 23:59:44.143716427 (14h20m18.658819013s) [upload=14749]: Upload finished. err: <nil> Aug 8 00:00:34.719721220 (14h21m9.234823804s) [upload=14759]: Starting upload Aug 8 00:00:34.719816520 (14h21m9.2349191s) [upload=14759]: Done starting upload Aug 8 00:00:34.719975322 (14h21m9.235077899s) [upload=14759]: Uploading segments... Aug 8 00:00:34.720108487 (14h21m9.235211073s) [upload=14759]: Got next segment. Inline: true Aug 8 00:00:34.720116703 (14h21m9.235219288s) [upload=14759]: Waiting for error group managing segments... Aug 8 00:00:34.720194402 (14h21m9.235296985s) [upload=14759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:34.720201518 (14h21m9.235304101s) [upload=14759]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:34.720205160 (14h21m9.23530774s) [upload=14759]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:34.794300564 (14h21m9.309403157s) [upload=14759]: Done uploading segments... Aug 8 00:00:34.794338844 (14h21m9.309441422s) [upload=14759]: Upload finished. err: <nil> Aug 8 00:01:01.776429007 (14h21m36.291531596s) [upload=14768]: Starting upload Aug 8 00:01:01.776506683 (14h21m36.291609246s) [upload=14768]: Done starting upload Aug 8 00:01:01.776613033 (14h21m36.291715611s) [upload=14768]: Uploading segments... Aug 8 00:01:01.776661337 (14h21m36.291763928s) [upload=14768]: Got next segment. Inline: true Aug 8 00:01:01.776666749 (14h21m36.29176933s) [upload=14768]: Waiting for error group managing segments... Aug 8 00:01:01.776715825 (14h21m36.291818408s) [upload=14768]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:01.776720751 (14h21m36.291823332s) [upload=14768]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:01.776724804 (14h21m36.291827383s) [upload=14768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:31.728405977 (14h22m6.243508555s) [upload=14768]: Done uploading segments... Aug 8 00:01:31.728424052 (14h22m6.243526628s) [upload=14768]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:02:03.679985597 (14h22m38.19508816s) [upload=14786]: Starting upload Aug 8 00:02:03.680087933 (14h22m38.19519052s) [upload=14786]: Done starting upload Aug 8 00:02:03.680194558 (14h22m38.195297135s) [upload=14786]: Uploading segments... Aug 8 00:02:03.680249278 (14h22m38.195351863s) [upload=14786]: Got next segment. Inline: true Aug 8 00:02:03.680254458 (14h22m38.195357034s) [upload=14786]: Waiting for error group managing segments... Aug 8 00:02:03.680309917 (14h22m38.195412483s) [upload=14786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:03.680314576 (14h22m38.195417152s) [upload=14786]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:03.680318116 (14h22m38.195420692s) [upload=14786]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:03.773554102 (14h22m38.288656668s) [upload=14786]: Done uploading segments... Aug 8 00:02:03.773587490 (14h22m38.288690069s) [upload=14786]: Upload finished. err: <nil> Aug 8 00:02:50.218217757 (14h23m24.733320413s) [upload=14799]: Starting upload Aug 8 00:02:50.218299459 (14h23m24.733402029s) [upload=14799]: Done starting upload Aug 8 00:02:50.218366696 (14h23m24.733469274s) [upload=14799]: Uploading segments... Aug 8 00:02:50.218404998 (14h23m24.733507583s) [upload=14799]: Got next segment. Inline: true Aug 8 00:02:50.218409405 (14h23m24.733511982s) [upload=14799]: Waiting for error group managing segments... Aug 8 00:02:50.218452564 (14h23m24.733555139s) [upload=14799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:50.218457853 (14h23m24.733560433s) [upload=14799]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:50.218462652 (14h23m24.733565229s) [upload=14799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:50.299378321 (14h23m24.814480907s) [upload=14799]: Done uploading segments... Aug 8 00:02:50.299417555 (14h23m24.814520132s) [upload=14799]: Upload finished. err: <nil> Aug 8 00:03:43.640902837 (14h24m18.156005417s) [upload=14815]: Starting upload Aug 8 00:03:43.640984215 (14h24m18.1560868s) [upload=14815]: Done starting upload Aug 8 00:03:43.641134510 (14h24m18.156237076s) [upload=14815]: Uploading segments... Aug 8 00:03:43.641193969 (14h24m18.156296557s) [upload=14815]: Got next segment. Inline: true Aug 8 00:03:43.641200481 (14h24m18.156303062s) [upload=14815]: Waiting for error group managing segments... Aug 8 00:03:43.641360742 (14h24m18.156463325s) [upload=14815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:03:43.641376086 (14h24m18.156478665s) [upload=14815]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:03:43.641379649 (14h24m18.156482228s) [upload=14815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:04:13.608341576 (14h24m48.123444144s) [upload=14815]: Done uploading segments... Aug 8 00:04:13.608361002 (14h24m48.123463569s) [upload=14815]: Upload finished. err: metaclient: context deadline exceeded Duration: 204.059099ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [....................................X..XXX..X...X...........] duration: 199.342006ms FAIL (2025-08-08 00:08:47 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 199.372299ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 1.197804474s FAIL (2025-08-08 00:08:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== piece... Aug 8 00:06:47.146014029 (14h27m21.661116615s) [noise=true , address=104.36.82.23:4456 , piece_id=6NTCQI462IGFCUSV6ZV2QVB2JB6XSIMBH54ABTWGOFPVNK74ON5Q, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.146042952 (14h27m21.661145531s) [noise=true , address=79.127.205.233:4636 , piece_id=XSBDMHKCPCIIQOV7SMEHSU5USOTUTVLNE2K4VHTI2O44IV5EGQBQ, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.146392229 (14h27m21.661494798s) [noise=true , address=104.36.82.9:4010 , piece_id=3MZMFKQ2LIPOSVDHEMGJHF5O3K7US3LKYSEGXPTKZMWDYMDKAOSQ, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.146489907 (14h27m21.661592493s) [noise=true , address=104.36.82.44:4883 , piece_id=CIX3VQVGNNZ2SSQYPXNZA3QZPQILYDXWGB66D3XMV3XZFG5IJDAA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.146751379 (14h27m21.661853965s) [noise=true , address=64.49.13.137:4494 , piece_id=XHXS6IICALFTEEQ6L3ARHBJMAQ6Z7AFIUIWWKSLCASRZVSY5IM2A, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.146970578 (14h27m21.662073166s) [noise=true , address=104.36.82.43:4868 , piece_id=7ZBUMTM2KGHZVQGM44SPYHZIK6BYKZ4RJLQNGEOLXK5FIBDOQQJA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.147102314 (14h27m21.662204899s) [noise=true , address=45.91.116.8:4065 , piece_id=FRBY35WXD6GF4WVNQXC5U4BDFGAKMCACN3GYCQYM3KIU7VR5F7EQ, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.147440038 (14h27m21.662542625s) [noise=true , address=104.36.82.13:4195 , piece_id=SE3RX356M46XAAHYGXSM56S4KC4QVSR4S3WPHHMLJHL2Y2U5RLOQ, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.147464022 (14h27m21.662566606s) [noise=true , address=64.49.13.27:4968 , piece_id=CG322ANVVFEA6XYPOW3FAJMWAHM3GJ72LCWWJMLMR6WZP2JIYM7A, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.147806500 (14h27m21.662909085s) [noise=true , address=104.36.82.14:4231 , piece_id=MDMME3XQYMMMTY2D2AO4JB2SHPJ6A4UPWN7DWXVPRI4EUAALNNBA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.148072099 (14h27m21.663174685s) [noise=true , address=104.36.82.60:4346 , piece_id=6ZKPSHS3NT77FZPZSN2BPX5A2T62JQJRJKD3H25QVZEPB34YDY3A, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.148231380 (14h27m21.663333959s) [noise=true , address=77.90.31.11:4557 , piece_id=VX6K65NE4IXJGBJEAC6GWXVMVFMOWJJJKUKWZSSG6XNCNFHH4IHA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.148503333 (14h27m21.663605919s) [noise=true , address=104.36.82.54:4163 , piece_id=5BYDGKBFV6DAPJOBSY52IWQ5HHLF6JPZ7PN6R7TY26KVL6CGI46Q, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.151364728 (14h27m21.666467312s) [noise=true , address=104.36.82.22:4511 , piece_id=DH5JH2TMIJEM7QXT3DIFIYQOOXAJNRUZAQD6QP7ZHUBZARBIXASA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.151909137 (14h27m21.667011732s) [noise=true , address=104.36.82.47:4527 , piece_id=LMI25BZRKDKTM442UA3DBFAFOI45RNSBXNS6K3J63GZKNFCIKLMQ, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.152324438 (14h27m21.667427024s) [noise=true , address=104.36.82.62:4416 , piece_id=C3BGVDY6FCK2WXISID4FQYE2XR3Q2SW5N5DETAUNVEZAFOT7MYXA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.269249030 (14h27m21.7843516s) [upload=14868]: Next returned nil segment Aug 8 00:06:47.269265635 (14h27m21.784368211s) [upload=14868]: Waiting for error group managing segments... Aug 8 00:06:47.280030107 (14h27m21.795132688s) [noise=true , address=104.36.82.62:4416 , piece_id=C3BGVDY6FCK2WXISID4FQYE2XR3Q2SW5N5DETAUNVEZAFOT7MYXA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.282243133 (14h27m21.797345711s) [noise=true , address=104.36.82.43:4868 , piece_id=7ZBUMTM2KGHZVQGM44SPYHZIK6BYKZ4RJLQNGEOLXK5FIBDOQQJA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.282935406 (14h27m21.798037984s) [noise=true , address=104.36.82.9:4010 , piece_id=3MZMFKQ2LIPOSVDHEMGJHF5O3K7US3LKYSEGXPTKZMWDYMDKAOSQ, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.283797129 (14h27m21.798899704s) [noise=true , address=104.36.82.13:4195 , piece_id=SE3RX356M46XAAHYGXSM56S4KC4QVSR4S3WPHHMLJHL2Y2U5RLOQ, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.288848212 (14h27m21.803950789s) [noise=true , address=45.91.116.5:4301 , piece_id=GQLYE5Z2Q2SHC2WIEZMTPRGYQ7B4EQZZYJLVC4NQMRENIDZATA5Q, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.292885927 (14h27m21.807988496s) [noise=true , address=45.91.116.8:4065 , piece_id=FRBY35WXD6GF4WVNQXC5U4BDFGAKMCACN3GYCQYM3KIU7VR5F7EQ, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.296224440 (14h27m21.811327016s) [noise=true , address=104.36.82.14:4231 , piece_id=MDMME3XQYMMMTY2D2AO4JB2SHPJ6A4UPWN7DWXVPRI4EUAALNNBA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.299834425 (14h27m21.81493699s) [noise=true , address=104.36.82.54:4163 , piece_id=5BYDGKBFV6DAPJOBSY52IWQ5HHLF6JPZ7PN6R7TY26KVL6CGI46Q, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.305883863 (14h27m21.820986439s) [noise=true , address=104.36.82.23:4456 , piece_id=6NTCQI462IGFCUSV6ZV2QVB2JB6XSIMBH54ABTWGOFPVNK74ON5Q, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.309497417 (14h27m21.824599987s) [noise=true , address=104.36.82.47:4527 , piece_id=LMI25BZRKDKTM442UA3DBFAFOI45RNSBXNS6K3J63GZKNFCIKLMQ, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.323758805 (14h27m21.838861383s) [noise=true , address=77.90.31.11:4557 , piece_id=VX6K65NE4IXJGBJEAC6GWXVMVFMOWJJJKUKWZSSG6XNCNFHH4IHA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.326822007 (14h27m21.841924593s) [noise=true , address=104.36.82.45:4938 , piece_id=QI275JMAE4DVWGO5CY3IDTC6WKZCNPPCO3LKE5EK74TGJSBLB2VA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.327668872 (14h27m21.842771449s) [noise=true , address=104.36.82.22:4511 , piece_id=DH5JH2TMIJEM7QXT3DIFIYQOOXAJNRUZAQD6QP7ZHUBZARBIXASA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.339100720 (14h27m21.854203297s) [noise=true , address=104.36.82.44:4883 , piece_id=CIX3VQVGNNZ2SSQYPXNZA3QZPQILYDXWGB66D3XMV3XZFG5IJDAA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.339134259 (14h27m21.854236839s) [seg_pos={0 0}, upload=14868]: Segment reached optimal threshold of 14 pieces. Aug 8 00:06:47.339343619 (14h27m21.854446181s) [noise=true , address=104.36.82.60:4346 , piece_id=6ZKPSHS3NT77FZPZSN2BPX5A2T62JQJRJKD3H25QVZEPB34YDY3A, seg_pos={0 0}, upload=14868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72): protocol: expected piece hash; context canceled; EOF Aug 8 00:06:47.339560288 (14h27m21.854662868s) [noise=true , address=79.127.205.233:4636 , piece_id=XSBDMHKCPCIIQOV7SMEHSU5USOTUTVLNE2K4VHTI2O44IV5EGQBQ, seg_pos={0 0}, upload=14868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mTJmjiHwMJqSkuWKrFnAgJUt1gjQ4wPjfgY7qouWcanaNUy46): protocol: read tcp 159.223.143.39:35278->79.127.205.233:4636: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:06:47.339701256 (14h27m21.854803837s) [noise=true , address=64.49.13.137:4494 , piece_id=XHXS6IICALFTEEQ6L3ARHBJMAQ6Z7AFIUIWWKSLCASRZVSY5IM2A, seg_pos={0 0}, upload=14868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TX66fDVv3qu4yBGx9FNZTiArNeUnqxjMYwb3VGqYCQRe8RwoU): protocol: read tcp 159.223.143.39:57032->64.49.13.137:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:06:47.339837792 (14h27m21.854940379s) [noise=true , address=64.49.13.27:4968 , piece_id=CG322ANVVFEA6XYPOW3FAJMWAHM3GJ72LCWWJMLMR6WZP2JIYM7A, seg_pos={0 0}, upload=14868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk): protocol: read tcp 159.223.143.39:48324->64.49.13.27:4968: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Aug 8 00:06:47.339968534 (14h27m21.855071118s) [seg_pos={0 0}, upload=14868]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:06:47.340079739 (14h27m21.855182316s) [upload=14868]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:06:47.340086599 (14h27m21.855189182s) [upload=14868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:06:47.400977058 (14h27m21.916079641s) [upload=14868]: Done uploading segments... Aug 8 00:06:47.401023653 (14h27m21.916126232s) [upload=14868]: Upload finished. err: <nil> Aug 8 00:07:57.606754320 (14h28m32.121856897s) [upload=14881]: Starting upload Aug 8 00:07:57.606822750 (14h28m32.121925318s) [upload=14881]: Done starting upload Aug 8 00:07:57.607097187 (14h28m32.122199756s) [upload=14881]: Uploading segments... Aug 8 00:07:57.607354625 (14h28m32.122457207s) [upload=14881]: Got next segment. Inline: false Aug 8 00:07:57.607435490 (14h28m32.122538073s) [upload=14881]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:07:57.607441733 (14h28m32.122544313s) [upload=14881]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:07:57.622249420 (14h28m32.137352005s) [upload=14881]: Next segment err: metaclient: Too Many Requests Aug 8 00:07:57.622307892 (14h28m32.137410469s) [upload=14881]: Done uploading segments... Aug 8 00:07:57.622322946 (14h28m32.137425515s) [upload=14881]: Upload finished. err: metaclient: Too Many Requests Duration: 1.19782989s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X..........X............] duration: 771.332339ms FAIL (2025-08-08 00:08:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 771.357097ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 8 out of 60 is failed (threshold=3) [.........................................XXXXXXX.X..........] duration: 212.985719ms FAIL (2025-08-08 00:08:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ne starting upload Aug 7 23:52:04.519569621 (14h12m39.034672186s) [upload=14619]: Uploading segments... Aug 7 23:52:04.519675504 (14h12m39.034778091s) [upload=14619]: Got next segment. Inline: true Aug 7 23:52:04.519701072 (14h12m39.034803649s) [upload=14619]: Waiting for error group managing segments... Aug 7 23:52:04.519755555 (14h12m39.034858131s) [upload=14619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:04.519760650 (14h12m39.034863226s) [upload=14619]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:04.519777078 (14h12m39.034879654s) [upload=14619]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:04.588422656 (14h12m39.103525241s) [upload=14619]: Done uploading segments... Aug 7 23:52:04.588444719 (14h12m39.103547297s) [upload=14619]: Upload finished. err: <nil> Aug 7 23:53:08.147087231 (14h13m42.662189804s) [upload=14641]: Starting upload Aug 7 23:53:08.147181411 (14h13m42.662283996s) [upload=14641]: Done starting upload Aug 7 23:53:08.147281202 (14h13m42.66238377s) [upload=14641]: Uploading segments... Aug 7 23:53:08.147322830 (14h13m42.662425415s) [upload=14641]: Got next segment. Inline: true Aug 7 23:53:08.147327701 (14h13m42.662430277s) [upload=14641]: Waiting for error group managing segments... Aug 7 23:53:08.147455761 (14h13m42.66255833s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:08.147460709 (14h13m42.662563285s) [upload=14641]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:08.147463890 (14h13m42.662566457s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:08.116969456 (14h14m42.632072039s) [upload=14641]: Done uploading segments... Aug 7 23:54:08.117007036 (14h14m42.632109618s) [upload=14641]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:22.831162336 (14h14m57.346264913s) [upload=14661]: Starting upload Aug 7 23:54:22.831234431 (14h14m57.346337008s) [upload=14661]: Done starting upload Aug 7 23:54:22.831306794 (14h14m57.346409362s) [upload=14661]: Uploading segments... Aug 7 23:54:22.831343257 (14h14m57.346445842s) [upload=14661]: Got next segment. Inline: true Aug 7 23:54:22.831347808 (14h14m57.346450375s) [upload=14661]: Waiting for error group managing segments... Aug 7 23:54:22.831399492 (14h14m57.346502071s) [upload=14661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:22.831404349 (14h14m57.346506915s) [upload=14661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:22.831407696 (14h14m57.346510265s) [upload=14661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:22.799123829 (14h15m57.314226409s) [upload=14661]: Done uploading segments... Aug 7 23:55:22.799154191 (14h15m57.314256772s) [upload=14661]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:22.918485444 (14h15m57.433588022s) [upload=14677]: Starting upload Aug 7 23:55:22.918566621 (14h15m57.433672141s) [upload=14677]: Done starting upload Aug 7 23:55:22.918647605 (14h15m57.433750172s) [upload=14677]: Uploading segments... Aug 7 23:55:22.918705315 (14h15m57.4338079s) [upload=14677]: Got next segment. Inline: true Aug 7 23:55:22.918710403 (14h15m57.43381298s) [upload=14677]: Waiting for error group managing segments... Aug 7 23:55:22.918764176 (14h15m57.433866753s) [upload=14677]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:22.918769239 (14h15m57.433871815s) [upload=14677]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:22.918772352 (14h15m57.433874928s) [upload=14677]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:22.891103920 (14h16m57.406206502s) [upload=14677]: Done uploading segments... Aug 7 23:56:22.891142892 (14h16m57.406245471s) [upload=14677]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:23.012512156 (14h16m57.527614733s) [upload=14690]: Starting upload Aug 7 23:56:23.012570683 (14h16m57.527673265s) [upload=14690]: Done starting upload Aug 7 23:56:23.012632890 (14h16m57.527735455s) [upload=14690]: Uploading segments... Aug 7 23:56:23.012675514 (14h16m57.527778093s) [upload=14690]: Got next segment. Inline: true Aug 7 23:56:23.012680516 (14h16m57.527783092s) [upload=14690]: Waiting for error group managing segments... Aug 7 23:56:23.012734720 (14h16m57.527837295s) [upload=14690]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:23.012739356 (14h16m57.527841923s) [upload=14690]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:23.012742484 (14h16m57.527845059s) [upload=14690]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:22.981126119 (14h17m57.496228686s) [upload=14690]: Done uploading segments... Aug 7 23:57:22.981169317 (14h17m57.496271889s) [upload=14690]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:23.078169780 (14h17m57.593272348s) [upload=14705]: Starting upload Aug 7 23:57:23.078287503 (14h17m57.593390079s) [upload=14705]: Done starting upload Aug 7 23:57:23.078353530 (14h17m57.593456107s) [upload=14705]: Uploading segments... Aug 7 23:57:23.078394403 (14h17m57.593496992s) [upload=14705]: Got next segment. Inline: true Aug 7 23:57:23.078400888 (14h17m57.593503457s) [upload=14705]: Waiting for error group managing segments... Aug 7 23:57:23.078452630 (14h17m57.593555201s) [upload=14705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:23.078457270 (14h17m57.593559838s) [upload=14705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:23.078460483 (14h17m57.593563048s) [upload=14705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:23.049396934 (14h18m57.564499518s) [upload=14705]: Done uploading segments... Aug 7 23:58:23.049418059 (14h18m57.564520634s) [upload=14705]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:23.169720285 (14h18m57.684822852s) [upload=14722]: Starting upload Aug 7 23:58:23.169779212 (14h18m57.684881788s) [upload=14722]: Done starting upload Aug 7 23:58:23.169840834 (14h18m57.684943401s) [upload=14722]: Uploading segments... Aug 7 23:58:23.169880437 (14h18m57.684983021s) [upload=14722]: Got next segment. Inline: true Aug 7 23:58:23.169885772 (14h18m57.684988338s) [upload=14722]: Waiting for error group managing segments... Aug 7 23:58:23.169927728 (14h18m57.685030303s) [upload=14722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:23.169933919 (14h18m57.685036498s) [upload=14722]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:23.169938117 (14h18m57.685040704s) [upload=14722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:23.138140642 (14h19m57.653243209s) [upload=14722]: Done uploading segments... Aug 7 23:59:23.138159469 (14h19m57.653262035s) [upload=14722]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:35.502179618 (14h20m10.017282184s) [upload=14745]: Starting upload Aug 7 23:59:35.502246881 (14h20m10.017349449s) [upload=14745]: Done starting upload Aug 7 23:59:35.502308502 (14h20m10.017411071s) [upload=14745]: Uploading segments... Aug 7 23:59:35.502347394 (14h20m10.017449979s) [upload=14745]: Got next segment. Inline: true Aug 7 23:59:35.502351750 (14h20m10.017454327s) [upload=14745]: Waiting for error group managing segments... Aug 7 23:59:35.502390443 (14h20m10.017493009s) [upload=14745]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:35.502394821 (14h20m10.017497398s) [upload=14745]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:35.502398119 (14h20m10.017500695s) [upload=14745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:35.472447782 (14h21m9.987550364s) [upload=14745]: Done uploading segments... Aug 8 00:00:35.472473966 (14h21m9.987576558s) [upload=14745]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:00:35.593243192 (14h21m10.10834576s) [upload=14760]: Starting upload Aug 8 00:00:35.593395835 (14h21m10.108498415s) [upload=14760]: Done starting upload Aug 8 00:00:35.593497926 (14h21m10.10860049s) [upload=14760]: Uploading segments... Aug 8 00:00:35.593659481 (14h21m10.108762045s) [upload=14760]: Got next segment. Inline: true Aug 8 00:00:35.593667686 (14h21m10.108770273s) [upload=14760]: Waiting for error group managing segments... Aug 8 00:00:35.593750345 (14h21m10.10885293s) [upload=14760]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:35.593773930 (14h21m10.108876508s) [upload=14760]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:35.593782409 (14h21m10.108884991s) [upload=14760]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:35.662793141 (14h21m10.177895709s) [upload=14760]: Done uploading segments... Aug 8 00:00:35.662815087 (14h21m10.177917658s) [upload=14760]: Upload finished. err: <nil> Aug 8 00:01:23.648690375 (14h21m58.163792959s) [upload=14775]: Starting upload Aug 8 00:01:23.648799550 (14h21m58.163902126s) [upload=14775]: Done starting upload Aug 8 00:01:23.648928369 (14h21m58.164030934s) [upload=14775]: Uploading segments... Aug 8 00:01:23.648986999 (14h21m58.164089578s) [upload=14775]: Got next segment. Inline: true Aug 8 00:01:23.648992059 (14h21m58.164094634s) [upload=14775]: Waiting for error group managing segments... Aug 8 00:01:23.649037148 (14h21m58.164139714s) [upload=14775]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:23.649041722 (14h21m58.164144298s) [upload=14775]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:23.649044950 (14h21m58.164147527s) [upload=14775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:23.618855090 (14h22m58.133957671s) [upload=14775]: Done uploading segments... Aug 8 00:02:23.618900564 (14h22m58.134003142s) [upload=14775]: Upload finished. err: metaclient: context deadline exceeded Duration: 213.014082ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....................................XXX.X...XX.XX..........] duration: 458.959859ms FAIL (2025-08-08 00:08:36 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 0.31.10:4517 , piece_id=4XNCSILTEXFXI3T5YJLDOHWW2766ZY2YM723VBVF7XYPPTZWBKEA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.130393931 (14h19m30.645496508s) [noise=true , address=45.91.116.8:4048 , piece_id=DGULAYZHOXDN4IO3EHSUHYN3Z3GOMWRPSF3IF762Z7PQNDBWWUCA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.130898433 (14h19m30.646001001s) [noise=true , address=104.36.82.11:4115 , piece_id=BKAK43Z32DE3HOXYPSW4BH5LUWMQ3TINDTPESBBXOBHA6NDDCQSA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.131654285 (14h19m30.646756854s) [noise=true , address=77.90.31.16:4733 , piece_id=R6AKPWEKFLEI7NP4XAMYL72SHIKNIORWP4JXFZXLVRTVIOXWGF2Q, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.132299167 (14h19m30.647401729s) [noise=true , address=45.91.116.13:4153 , piece_id=G35ZWVBI2OZYXYXOY5RKDG6IO2G4PET4BDFHGLQV4NVTUVAGVNJQ, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.132815896 (14h19m30.647918459s) [noise=true , address=77.90.31.17:4772 , piece_id=76IYY7NQGZ4WZTPRZ5MW7B7ABMNNQJGHKGZHPKLCX4KHAWNJ7XGA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.133305230 (14h19m30.648407796s) [noise=true , address=104.36.82.6:4875 , piece_id=RHXGB245P3CNRWQJPKBZ4RL6BV5AGIHV35TVLDCWIUVCVU2TRZJA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.134064921 (14h19m30.649167503s) [noise=true , address=104.36.82.2:4731 , piece_id=374ZMGN47ZFQDXM2NI52ZLYMNYNUMB5HD7UZWKFGEKMDDAG4DUAA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.134501761 (14h19m30.649604335s) [noise=true , address=104.36.82.5:4840 , piece_id=JKJE7FIIKVDBL2MQF7IWVNMYH63QV72QXJOYARECFS5FJMSAOK4A, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.135039838 (14h19m30.650142405s) [noise=true , address=45.91.116.14:4191 , piece_id=5SWPQ2TBXYVYNFOGMH6QVANJVGNFEVOJ6UFMAX74DCC2WWKRSBNA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.135450095 (14h19m30.650552665s) [noise=true , address=77.90.31.19:4842 , piece_id=OUTUWUFXDDTLL52UJ6RTLORFEPWMQVL6JYBP3PPFY2DHS7JQHMBQ, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.135750292 (14h19m30.650852861s) [noise=true , address=77.90.31.15:4697 , piece_id=7MTL64JADCQRGS6UHDTT2ZGJFJV26TP4VYEPVM43MFLNXPPHFRQA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.136062113 (14h19m30.651164675s) [noise=true , address=104.36.82.9:4005 , piece_id=BJR6LEV57IPYIKASKNYGCMIKM7XMPZ6J3ASRB4B2ZZHAD3WQB47Q, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.136413662 (14h19m30.65151623s) [noise=true , address=45.91.116.9:4263 , piece_id=3G4QVYQG6AJ555B632PNRZWUUJM23K3GWSIPGNWJBCQR3ZZGZ4BQ, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.136771535 (14h19m30.651874107s) [noise=true , address=104.36.82.16:4295 , piece_id=QFJ444VYPVNGDMZ2WO2MBWMUH2U6M5K6TQ66T6SHQJW2CM245BTA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.209688234 (14h19m30.724790803s) [upload=14734]: Next returned nil segment Aug 7 23:58:56.209703014 (14h19m30.72480559s) [upload=14734]: Waiting for error group managing segments... Aug 7 23:58:56.214389537 (14h19m30.729492107s) [noise=true , address=45.91.116.9:4263 , piece_id=3G4QVYQG6AJ555B632PNRZWUUJM23K3GWSIPGNWJBCQR3ZZGZ4BQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.214445477 (14h19m30.729548054s) [noise=true , address=45.91.116.13:4153 , piece_id=G35ZWVBI2OZYXYXOY5RKDG6IO2G4PET4BDFHGLQV4NVTUVAGVNJQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.214979544 (14h19m30.73008212s) [noise=true , address=77.90.31.19:4842 , piece_id=OUTUWUFXDDTLL52UJ6RTLORFEPWMQVL6JYBP3PPFY2DHS7JQHMBQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.215082462 (14h19m30.730185028s) [noise=true , address=45.91.116.14:4191 , piece_id=5SWPQ2TBXYVYNFOGMH6QVANJVGNFEVOJ6UFMAX74DCC2WWKRSBNA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.215826575 (14h19m30.730929141s) [noise=true , address=77.90.31.10:4517 , piece_id=4XNCSILTEXFXI3T5YJLDOHWW2766ZY2YM723VBVF7XYPPTZWBKEA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.215911546 (14h19m30.731014112s) [noise=true , address=77.90.31.16:4733 , piece_id=R6AKPWEKFLEI7NP4XAMYL72SHIKNIORWP4JXFZXLVRTVIOXWGF2Q, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.216710632 (14h19m30.731813201s) [noise=true , address=77.90.31.18:4807 , piece_id=B3PSWCRJNGDMD6LDDM436N3X75NAA4DEEPLDKL2GTBBJBG3I6ZSQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.217564442 (14h19m30.732667019s) [noise=true , address=77.90.31.15:4697 , piece_id=7MTL64JADCQRGS6UHDTT2ZGJFJV26TP4VYEPVM43MFLNXPPHFRQA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.218100336 (14h19m30.733202913s) [noise=true , address=77.90.31.17:4772 , piece_id=76IYY7NQGZ4WZTPRZ5MW7B7ABMNNQJGHKGZHPKLCX4KHAWNJ7XGA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.220843127 (14h19m30.735945696s) [noise=true , address=104.36.82.11:4115 , piece_id=BKAK43Z32DE3HOXYPSW4BH5LUWMQ3TINDTPESBBXOBHA6NDDCQSA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.221894363 (14h19m30.736996944s) [noise=true , address=104.36.82.6:4875 , piece_id=RHXGB245P3CNRWQJPKBZ4RL6BV5AGIHV35TVLDCWIUVCVU2TRZJA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.227829104 (14h19m30.742931668s) [noise=true , address=45.91.116.8:4048 , piece_id=DGULAYZHOXDN4IO3EHSUHYN3Z3GOMWRPSF3IF762Z7PQNDBWWUCA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.326406182 (14h19m30.841508752s) [noise=true , address=104.36.82.9:4005 , piece_id=BJR6LEV57IPYIKASKNYGCMIKM7XMPZ6J3ASRB4B2ZZHAD3WQB47Q, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.326453057 (14h19m30.841555634s) [seg_pos={0 0}, upload=14734]: Segment reached optimal threshold of 13 pieces. Aug 7 23:58:56.326683605 (14h19m30.841786189s) [noise=true , address=104.36.82.2:4731 , piece_id=374ZMGN47ZFQDXM2NI52ZLYMNYNUMB5HD7UZWKFGEKMDDAG4DUAA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Aug 7 23:58:56.326786377 (14h19m30.84188896s) [noise=true , address=104.36.82.5:4840 , piece_id=JKJE7FIIKVDBL2MQF7IWVNMYH63QV72QXJOYARECFS5FJMSAOK4A, seg_pos={0 0}, upload=14734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: write tcp 159.223.143.39:60496->104.36.82.5:4840: use of closed network connection; write tcp 159.223.143.39:60496->104.36.82.5:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60496->104.36.82.5:4840: use of closed network connection Aug 7 23:58:56.326888961 (14h19m30.841991543s) [noise=true , address=104.36.82.16:4295 , piece_id=QFJ444VYPVNGDMZ2WO2MBWMUH2U6M5K6TQ66T6SHQJW2CM245BTA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: read tcp 159.223.143.39:34618->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Aug 7 23:58:56.326913735 (14h19m30.842016315s) [seg_pos={0 0}, upload=14734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:58:56.326998778 (14h19m30.842101358s) [upload=14734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:58:56.327004732 (14h19m30.84210731s) [upload=14734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:56.375303093 (14h19m30.89040566s) [upload=14734]: Done uploading segments... Aug 7 23:58:56.375321956 (14h19m30.890424523s) [upload=14734]: Upload finished. err: <nil> Aug 7 23:59:26.492160234 (14h20m1.007262811s) [upload=14742]: Starting upload Aug 7 23:59:26.492250238 (14h20m1.007352804s) [upload=14742]: Done starting upload Aug 7 23:59:26.492323252 (14h20m1.00742583s) [upload=14742]: Uploading segments... Aug 7 23:59:26.492605785 (14h20m1.007708349s) [upload=14742]: Got next segment. Inline: false Aug 7 23:59:26.492681402 (14h20m1.007783987s) [upload=14742]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:26.492689520 (14h20m1.007792101s) [upload=14742]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:59:56.459090933 (14h20m30.974193512s) [upload=14742]: Next segment err: context deadline exceeded Aug 7 23:59:56.459165231 (14h20m30.974267796s) [upload=14742]: Done uploading segments... Aug 7 23:59:56.459181138 (14h20m30.974283703s) [upload=14742]: Upload finished. err: context deadline exceeded Aug 7 23:59:56.572495425 (14h20m31.087597994s) [upload=14751]: Starting upload Aug 7 23:59:56.572578962 (14h20m31.087681544s) [upload=14751]: Done starting upload Aug 7 23:59:56.572818459 (14h20m31.087921029s) [upload=14751]: Uploading segments... Aug 7 23:59:56.572953386 (14h20m31.088055953s) [upload=14751]: Got next segment. Inline: false Aug 7 23:59:56.573048904 (14h20m31.088151483s) [upload=14751]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:56.573066041 (14h20m31.088168618s) [upload=14751]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:00:26.529103664 (14h21m1.044206252s) [upload=14751]: Next segment err: context deadline exceeded Aug 8 00:00:26.529241226 (14h21m1.044343811s) [upload=14751]: Done uploading segments... Aug 8 00:00:26.529265086 (14h21m1.044367654s) [upload=14751]: Upload finished. err: context deadline exceeded Duration: 458.984885ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 6 out of 60 is failed (threshold=3) [......................................X.XX.X.....X..X.......] duration: 249.390459ms FAIL (2025-08-08 00:08:33 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Upload finished. err: <nil> Aug 7 23:56:22.913855555 (14h16m57.428958117s) [upload=14689]: Starting upload Aug 7 23:56:22.913925154 (14h16m57.429027734s) [upload=14689]: Done starting upload Aug 7 23:56:22.913999531 (14h16m57.429102093s) [upload=14689]: Uploading segments... Aug 7 23:56:22.914048699 (14h16m57.429151282s) [upload=14689]: Got next segment. Inline: true Aug 7 23:56:22.914052623 (14h16m57.4291552s) [upload=14689]: Waiting for error group managing segments... Aug 7 23:56:22.914089928 (14h16m57.429192503s) [upload=14689]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:22.914093434 (14h16m57.42919601s) [upload=14689]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:22.914096175 (14h16m57.429198751s) [upload=14689]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:52.884393262 (14h17m27.399495844s) [upload=14689]: Done uploading segments... Aug 7 23:56:52.884415558 (14h17m27.399518126s) [upload=14689]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:09.822697015 (14h17m44.337799585s) [upload=14701]: Starting upload Aug 7 23:57:09.822774891 (14h17m44.337877456s) [upload=14701]: Done starting upload Aug 7 23:57:09.822876390 (14h17m44.337978955s) [upload=14701]: Uploading segments... Aug 7 23:57:09.823345112 (14h17m44.338447694s) [upload=14701]: Got next segment. Inline: true Aug 7 23:57:09.823357163 (14h17m44.338459742s) [upload=14701]: Waiting for error group managing segments... Aug 7 23:57:09.823423761 (14h17m44.338526328s) [upload=14701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:09.823429578 (14h17m44.338532154s) [upload=14701]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:09.823433695 (14h17m44.338536271s) [upload=14701]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:09.886245350 (14h17m44.401347936s) [upload=14701]: Done uploading segments... Aug 7 23:57:09.886269151 (14h17m44.401371729s) [upload=14701]: Upload finished. err: <nil> Aug 7 23:58:11.646122622 (14h18m46.161225204s) [upload=14719]: Starting upload Aug 7 23:58:11.646203494 (14h18m46.16130607s) [upload=14719]: Done starting upload Aug 7 23:58:11.646278294 (14h18m46.161380873s) [upload=14719]: Uploading segments... Aug 7 23:58:11.646339731 (14h18m46.161442316s) [upload=14719]: Got next segment. Inline: true Aug 7 23:58:11.646344984 (14h18m46.16144756s) [upload=14719]: Waiting for error group managing segments... Aug 7 23:58:11.646388852 (14h18m46.161491429s) [upload=14719]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:11.646393482 (14h18m46.161496048s) [upload=14719]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:11.646396765 (14h18m46.161499342s) [upload=14719]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:11.705007157 (14h18m46.220109734s) [upload=14719]: Done uploading segments... Aug 7 23:58:11.705026176 (14h18m46.220128742s) [upload=14719]: Upload finished. err: <nil> Aug 7 23:58:41.852871585 (14h19m16.367974172s) [upload=14731]: Starting upload Aug 7 23:58:41.852974809 (14h19m16.368077394s) [upload=14731]: Done starting upload Aug 7 23:58:41.853095753 (14h19m16.368198318s) [upload=14731]: Uploading segments... Aug 7 23:58:41.853172118 (14h19m16.368274707s) [upload=14731]: Got next segment. Inline: true Aug 7 23:58:41.853178991 (14h19m16.368281577s) [upload=14731]: Waiting for error group managing segments... Aug 7 23:58:41.853256548 (14h19m16.368359135s) [upload=14731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:41.853263687 (14h19m16.368366273s) [upload=14731]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:41.853268996 (14h19m16.368371576s) [upload=14731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:41.919489485 (14h19m16.434592068s) [upload=14731]: Done uploading segments... Aug 7 23:58:41.919512553 (14h19m16.434615126s) [upload=14731]: Upload finished. err: <nil> Aug 7 23:59:44.072708375 (14h20m18.587810945s) [upload=14749]: Starting upload Aug 7 23:59:44.072763052 (14h20m18.587865629s) [upload=14749]: Done starting upload Aug 7 23:59:44.072818447 (14h20m18.587921015s) [upload=14749]: Uploading segments... Aug 7 23:59:44.072859036 (14h20m18.587961622s) [upload=14749]: Got next segment. Inline: true Aug 7 23:59:44.072863633 (14h20m18.587966209s) [upload=14749]: Waiting for error group managing segments... Aug 7 23:59:44.072902482 (14h20m18.588005058s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:44.072906670 (14h20m18.588009237s) [upload=14749]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:44.072910115 (14h20m18.588012692s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:44.143691065 (14h20m18.658793888s) [upload=14749]: Done uploading segments... Aug 7 23:59:44.143716427 (14h20m18.658819013s) [upload=14749]: Upload finished. err: <nil> Aug 8 00:00:34.719721220 (14h21m9.234823804s) [upload=14759]: Starting upload Aug 8 00:00:34.719816520 (14h21m9.2349191s) [upload=14759]: Done starting upload Aug 8 00:00:34.719975322 (14h21m9.235077899s) [upload=14759]: Uploading segments... Aug 8 00:00:34.720108487 (14h21m9.235211073s) [upload=14759]: Got next segment. Inline: true Aug 8 00:00:34.720116703 (14h21m9.235219288s) [upload=14759]: Waiting for error group managing segments... Aug 8 00:00:34.720194402 (14h21m9.235296985s) [upload=14759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:34.720201518 (14h21m9.235304101s) [upload=14759]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:34.720205160 (14h21m9.23530774s) [upload=14759]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:34.794300564 (14h21m9.309403157s) [upload=14759]: Done uploading segments... Aug 8 00:00:34.794338844 (14h21m9.309441422s) [upload=14759]: Upload finished. err: <nil> Aug 8 00:01:01.776429007 (14h21m36.291531596s) [upload=14768]: Starting upload Aug 8 00:01:01.776506683 (14h21m36.291609246s) [upload=14768]: Done starting upload Aug 8 00:01:01.776613033 (14h21m36.291715611s) [upload=14768]: Uploading segments... Aug 8 00:01:01.776661337 (14h21m36.291763928s) [upload=14768]: Got next segment. Inline: true Aug 8 00:01:01.776666749 (14h21m36.29176933s) [upload=14768]: Waiting for error group managing segments... Aug 8 00:01:01.776715825 (14h21m36.291818408s) [upload=14768]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:01.776720751 (14h21m36.291823332s) [upload=14768]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:01.776724804 (14h21m36.291827383s) [upload=14768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:31.728405977 (14h22m6.243508555s) [upload=14768]: Done uploading segments... Aug 8 00:01:31.728424052 (14h22m6.243526628s) [upload=14768]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:02:03.679985597 (14h22m38.19508816s) [upload=14786]: Starting upload Aug 8 00:02:03.680087933 (14h22m38.19519052s) [upload=14786]: Done starting upload Aug 8 00:02:03.680194558 (14h22m38.195297135s) [upload=14786]: Uploading segments... Aug 8 00:02:03.680249278 (14h22m38.195351863s) [upload=14786]: Got next segment. Inline: true Aug 8 00:02:03.680254458 (14h22m38.195357034s) [upload=14786]: Waiting for error group managing segments... Aug 8 00:02:03.680309917 (14h22m38.195412483s) [upload=14786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:03.680314576 (14h22m38.195417152s) [upload=14786]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:03.680318116 (14h22m38.195420692s) [upload=14786]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:03.773554102 (14h22m38.288656668s) [upload=14786]: Done uploading segments... Aug 8 00:02:03.773587490 (14h22m38.288690069s) [upload=14786]: Upload finished. err: <nil> Aug 8 00:02:50.218217757 (14h23m24.733320413s) [upload=14799]: Starting upload Aug 8 00:02:50.218299459 (14h23m24.733402029s) [upload=14799]: Done starting upload Aug 8 00:02:50.218366696 (14h23m24.733469274s) [upload=14799]: Uploading segments... Aug 8 00:02:50.218404998 (14h23m24.733507583s) [upload=14799]: Got next segment. Inline: true Aug 8 00:02:50.218409405 (14h23m24.733511982s) [upload=14799]: Waiting for error group managing segments... Aug 8 00:02:50.218452564 (14h23m24.733555139s) [upload=14799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:50.218457853 (14h23m24.733560433s) [upload=14799]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:50.218462652 (14h23m24.733565229s) [upload=14799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:50.299378321 (14h23m24.814480907s) [upload=14799]: Done uploading segments... Aug 8 00:02:50.299417555 (14h23m24.814520132s) [upload=14799]: Upload finished. err: <nil> Aug 8 00:03:43.640902837 (14h24m18.156005417s) [upload=14815]: Starting upload Aug 8 00:03:43.640984215 (14h24m18.1560868s) [upload=14815]: Done starting upload Aug 8 00:03:43.641134510 (14h24m18.156237076s) [upload=14815]: Uploading segments... Aug 8 00:03:43.641193969 (14h24m18.156296557s) [upload=14815]: Got next segment. Inline: true Aug 8 00:03:43.641200481 (14h24m18.156303062s) [upload=14815]: Waiting for error group managing segments... Aug 8 00:03:43.641360742 (14h24m18.156463325s) [upload=14815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:03:43.641376086 (14h24m18.156478665s) [upload=14815]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:03:43.641379649 (14h24m18.156482228s) [upload=14815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:04:13.608341576 (14h24m48.123444144s) [upload=14815]: Done uploading segments... Aug 8 00:04:13.608361002 (14h24m48.123463569s) [upload=14815]: Upload finished. err: metaclient: context deadline exceeded Duration: 249.416561ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [XX.............................................X............] duration: 2.229110267s FAIL (2025-08-08 00:08:31 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.229139363s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...........................................XX..XX...........] duration: 1.009056163s FAIL (2025-08-08 00:08:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.009121803s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 6 out of 60 is failed (threshold=3) [.......................................X.XX.X.....X..X......] duration: 285.020792ms FAIL (2025-08-08 00:08:18 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Upload finished. err: <nil> Aug 7 23:56:22.913855555 (14h16m57.428958117s) [upload=14689]: Starting upload Aug 7 23:56:22.913925154 (14h16m57.429027734s) [upload=14689]: Done starting upload Aug 7 23:56:22.913999531 (14h16m57.429102093s) [upload=14689]: Uploading segments... Aug 7 23:56:22.914048699 (14h16m57.429151282s) [upload=14689]: Got next segment. Inline: true Aug 7 23:56:22.914052623 (14h16m57.4291552s) [upload=14689]: Waiting for error group managing segments... Aug 7 23:56:22.914089928 (14h16m57.429192503s) [upload=14689]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:22.914093434 (14h16m57.42919601s) [upload=14689]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:22.914096175 (14h16m57.429198751s) [upload=14689]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:52.884393262 (14h17m27.399495844s) [upload=14689]: Done uploading segments... Aug 7 23:56:52.884415558 (14h17m27.399518126s) [upload=14689]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:09.822697015 (14h17m44.337799585s) [upload=14701]: Starting upload Aug 7 23:57:09.822774891 (14h17m44.337877456s) [upload=14701]: Done starting upload Aug 7 23:57:09.822876390 (14h17m44.337978955s) [upload=14701]: Uploading segments... Aug 7 23:57:09.823345112 (14h17m44.338447694s) [upload=14701]: Got next segment. Inline: true Aug 7 23:57:09.823357163 (14h17m44.338459742s) [upload=14701]: Waiting for error group managing segments... Aug 7 23:57:09.823423761 (14h17m44.338526328s) [upload=14701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:09.823429578 (14h17m44.338532154s) [upload=14701]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:09.823433695 (14h17m44.338536271s) [upload=14701]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:09.886245350 (14h17m44.401347936s) [upload=14701]: Done uploading segments... Aug 7 23:57:09.886269151 (14h17m44.401371729s) [upload=14701]: Upload finished. err: <nil> Aug 7 23:58:11.646122622 (14h18m46.161225204s) [upload=14719]: Starting upload Aug 7 23:58:11.646203494 (14h18m46.16130607s) [upload=14719]: Done starting upload Aug 7 23:58:11.646278294 (14h18m46.161380873s) [upload=14719]: Uploading segments... Aug 7 23:58:11.646339731 (14h18m46.161442316s) [upload=14719]: Got next segment. Inline: true Aug 7 23:58:11.646344984 (14h18m46.16144756s) [upload=14719]: Waiting for error group managing segments... Aug 7 23:58:11.646388852 (14h18m46.161491429s) [upload=14719]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:11.646393482 (14h18m46.161496048s) [upload=14719]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:11.646396765 (14h18m46.161499342s) [upload=14719]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:11.705007157 (14h18m46.220109734s) [upload=14719]: Done uploading segments... Aug 7 23:58:11.705026176 (14h18m46.220128742s) [upload=14719]: Upload finished. err: <nil> Aug 7 23:58:41.852871585 (14h19m16.367974172s) [upload=14731]: Starting upload Aug 7 23:58:41.852974809 (14h19m16.368077394s) [upload=14731]: Done starting upload Aug 7 23:58:41.853095753 (14h19m16.368198318s) [upload=14731]: Uploading segments... Aug 7 23:58:41.853172118 (14h19m16.368274707s) [upload=14731]: Got next segment. Inline: true Aug 7 23:58:41.853178991 (14h19m16.368281577s) [upload=14731]: Waiting for error group managing segments... Aug 7 23:58:41.853256548 (14h19m16.368359135s) [upload=14731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:41.853263687 (14h19m16.368366273s) [upload=14731]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:41.853268996 (14h19m16.368371576s) [upload=14731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:41.919489485 (14h19m16.434592068s) [upload=14731]: Done uploading segments... Aug 7 23:58:41.919512553 (14h19m16.434615126s) [upload=14731]: Upload finished. err: <nil> Aug 7 23:59:44.072708375 (14h20m18.587810945s) [upload=14749]: Starting upload Aug 7 23:59:44.072763052 (14h20m18.587865629s) [upload=14749]: Done starting upload Aug 7 23:59:44.072818447 (14h20m18.587921015s) [upload=14749]: Uploading segments... Aug 7 23:59:44.072859036 (14h20m18.587961622s) [upload=14749]: Got next segment. Inline: true Aug 7 23:59:44.072863633 (14h20m18.587966209s) [upload=14749]: Waiting for error group managing segments... Aug 7 23:59:44.072902482 (14h20m18.588005058s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:44.072906670 (14h20m18.588009237s) [upload=14749]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:44.072910115 (14h20m18.588012692s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:44.143691065 (14h20m18.658793888s) [upload=14749]: Done uploading segments... Aug 7 23:59:44.143716427 (14h20m18.658819013s) [upload=14749]: Upload finished. err: <nil> Aug 8 00:00:34.719721220 (14h21m9.234823804s) [upload=14759]: Starting upload Aug 8 00:00:34.719816520 (14h21m9.2349191s) [upload=14759]: Done starting upload Aug 8 00:00:34.719975322 (14h21m9.235077899s) [upload=14759]: Uploading segments... Aug 8 00:00:34.720108487 (14h21m9.235211073s) [upload=14759]: Got next segment. Inline: true Aug 8 00:00:34.720116703 (14h21m9.235219288s) [upload=14759]: Waiting for error group managing segments... Aug 8 00:00:34.720194402 (14h21m9.235296985s) [upload=14759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:34.720201518 (14h21m9.235304101s) [upload=14759]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:34.720205160 (14h21m9.23530774s) [upload=14759]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:34.794300564 (14h21m9.309403157s) [upload=14759]: Done uploading segments... Aug 8 00:00:34.794338844 (14h21m9.309441422s) [upload=14759]: Upload finished. err: <nil> Aug 8 00:01:01.776429007 (14h21m36.291531596s) [upload=14768]: Starting upload Aug 8 00:01:01.776506683 (14h21m36.291609246s) [upload=14768]: Done starting upload Aug 8 00:01:01.776613033 (14h21m36.291715611s) [upload=14768]: Uploading segments... Aug 8 00:01:01.776661337 (14h21m36.291763928s) [upload=14768]: Got next segment. Inline: true Aug 8 00:01:01.776666749 (14h21m36.29176933s) [upload=14768]: Waiting for error group managing segments... Aug 8 00:01:01.776715825 (14h21m36.291818408s) [upload=14768]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:01.776720751 (14h21m36.291823332s) [upload=14768]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:01.776724804 (14h21m36.291827383s) [upload=14768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:31.728405977 (14h22m6.243508555s) [upload=14768]: Done uploading segments... Aug 8 00:01:31.728424052 (14h22m6.243526628s) [upload=14768]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:02:03.679985597 (14h22m38.19508816s) [upload=14786]: Starting upload Aug 8 00:02:03.680087933 (14h22m38.19519052s) [upload=14786]: Done starting upload Aug 8 00:02:03.680194558 (14h22m38.195297135s) [upload=14786]: Uploading segments... Aug 8 00:02:03.680249278 (14h22m38.195351863s) [upload=14786]: Got next segment. Inline: true Aug 8 00:02:03.680254458 (14h22m38.195357034s) [upload=14786]: Waiting for error group managing segments... Aug 8 00:02:03.680309917 (14h22m38.195412483s) [upload=14786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:03.680314576 (14h22m38.195417152s) [upload=14786]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:03.680318116 (14h22m38.195420692s) [upload=14786]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:03.773554102 (14h22m38.288656668s) [upload=14786]: Done uploading segments... Aug 8 00:02:03.773587490 (14h22m38.288690069s) [upload=14786]: Upload finished. err: <nil> Aug 8 00:02:50.218217757 (14h23m24.733320413s) [upload=14799]: Starting upload Aug 8 00:02:50.218299459 (14h23m24.733402029s) [upload=14799]: Done starting upload Aug 8 00:02:50.218366696 (14h23m24.733469274s) [upload=14799]: Uploading segments... Aug 8 00:02:50.218404998 (14h23m24.733507583s) [upload=14799]: Got next segment. Inline: true Aug 8 00:02:50.218409405 (14h23m24.733511982s) [upload=14799]: Waiting for error group managing segments... Aug 8 00:02:50.218452564 (14h23m24.733555139s) [upload=14799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:50.218457853 (14h23m24.733560433s) [upload=14799]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:50.218462652 (14h23m24.733565229s) [upload=14799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:50.299378321 (14h23m24.814480907s) [upload=14799]: Done uploading segments... Aug 8 00:02:50.299417555 (14h23m24.814520132s) [upload=14799]: Upload finished. err: <nil> Aug 8 00:03:43.640902837 (14h24m18.156005417s) [upload=14815]: Starting upload Aug 8 00:03:43.640984215 (14h24m18.1560868s) [upload=14815]: Done starting upload Aug 8 00:03:43.641134510 (14h24m18.156237076s) [upload=14815]: Uploading segments... Aug 8 00:03:43.641193969 (14h24m18.156296557s) [upload=14815]: Got next segment. Inline: true Aug 8 00:03:43.641200481 (14h24m18.156303062s) [upload=14815]: Waiting for error group managing segments... Aug 8 00:03:43.641360742 (14h24m18.156463325s) [upload=14815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:03:43.641376086 (14h24m18.156478665s) [upload=14815]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:03:43.641379649 (14h24m18.156482228s) [upload=14815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:04:13.608341576 (14h24m48.123444144s) [upload=14815]: Done uploading segments... Aug 8 00:04:13.608361002 (14h24m48.123463569s) [upload=14815]: Upload finished. err: metaclient: context deadline exceeded Duration: 285.050268ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 4 out of 60 is failed (threshold=3) [XXX.............................................X...........] duration: 1.841399627s FAIL (2025-08-08 00:08:13 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.841432763s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [.....................................X..XXX..X...X..........] duration: 223.767127ms FAIL (2025-08-08 00:08:13 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 223.790615ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......................................XXX.X...XX.XX.........] duration: 557.666234ms FAIL (2025-08-08 00:08:10 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 0.31.10:4517 , piece_id=4XNCSILTEXFXI3T5YJLDOHWW2766ZY2YM723VBVF7XYPPTZWBKEA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.130393931 (14h19m30.645496508s) [noise=true , address=45.91.116.8:4048 , piece_id=DGULAYZHOXDN4IO3EHSUHYN3Z3GOMWRPSF3IF762Z7PQNDBWWUCA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.130898433 (14h19m30.646001001s) [noise=true , address=104.36.82.11:4115 , piece_id=BKAK43Z32DE3HOXYPSW4BH5LUWMQ3TINDTPESBBXOBHA6NDDCQSA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.131654285 (14h19m30.646756854s) [noise=true , address=77.90.31.16:4733 , piece_id=R6AKPWEKFLEI7NP4XAMYL72SHIKNIORWP4JXFZXLVRTVIOXWGF2Q, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.132299167 (14h19m30.647401729s) [noise=true , address=45.91.116.13:4153 , piece_id=G35ZWVBI2OZYXYXOY5RKDG6IO2G4PET4BDFHGLQV4NVTUVAGVNJQ, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.132815896 (14h19m30.647918459s) [noise=true , address=77.90.31.17:4772 , piece_id=76IYY7NQGZ4WZTPRZ5MW7B7ABMNNQJGHKGZHPKLCX4KHAWNJ7XGA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.133305230 (14h19m30.648407796s) [noise=true , address=104.36.82.6:4875 , piece_id=RHXGB245P3CNRWQJPKBZ4RL6BV5AGIHV35TVLDCWIUVCVU2TRZJA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.134064921 (14h19m30.649167503s) [noise=true , address=104.36.82.2:4731 , piece_id=374ZMGN47ZFQDXM2NI52ZLYMNYNUMB5HD7UZWKFGEKMDDAG4DUAA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.134501761 (14h19m30.649604335s) [noise=true , address=104.36.82.5:4840 , piece_id=JKJE7FIIKVDBL2MQF7IWVNMYH63QV72QXJOYARECFS5FJMSAOK4A, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.135039838 (14h19m30.650142405s) [noise=true , address=45.91.116.14:4191 , piece_id=5SWPQ2TBXYVYNFOGMH6QVANJVGNFEVOJ6UFMAX74DCC2WWKRSBNA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.135450095 (14h19m30.650552665s) [noise=true , address=77.90.31.19:4842 , piece_id=OUTUWUFXDDTLL52UJ6RTLORFEPWMQVL6JYBP3PPFY2DHS7JQHMBQ, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.135750292 (14h19m30.650852861s) [noise=true , address=77.90.31.15:4697 , piece_id=7MTL64JADCQRGS6UHDTT2ZGJFJV26TP4VYEPVM43MFLNXPPHFRQA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.136062113 (14h19m30.651164675s) [noise=true , address=104.36.82.9:4005 , piece_id=BJR6LEV57IPYIKASKNYGCMIKM7XMPZ6J3ASRB4B2ZZHAD3WQB47Q, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.136413662 (14h19m30.65151623s) [noise=true , address=45.91.116.9:4263 , piece_id=3G4QVYQG6AJ555B632PNRZWUUJM23K3GWSIPGNWJBCQR3ZZGZ4BQ, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.136771535 (14h19m30.651874107s) [noise=true , address=104.36.82.16:4295 , piece_id=QFJ444VYPVNGDMZ2WO2MBWMUH2U6M5K6TQ66T6SHQJW2CM245BTA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.209688234 (14h19m30.724790803s) [upload=14734]: Next returned nil segment Aug 7 23:58:56.209703014 (14h19m30.72480559s) [upload=14734]: Waiting for error group managing segments... Aug 7 23:58:56.214389537 (14h19m30.729492107s) [noise=true , address=45.91.116.9:4263 , piece_id=3G4QVYQG6AJ555B632PNRZWUUJM23K3GWSIPGNWJBCQR3ZZGZ4BQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.214445477 (14h19m30.729548054s) [noise=true , address=45.91.116.13:4153 , piece_id=G35ZWVBI2OZYXYXOY5RKDG6IO2G4PET4BDFHGLQV4NVTUVAGVNJQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.214979544 (14h19m30.73008212s) [noise=true , address=77.90.31.19:4842 , piece_id=OUTUWUFXDDTLL52UJ6RTLORFEPWMQVL6JYBP3PPFY2DHS7JQHMBQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.215082462 (14h19m30.730185028s) [noise=true , address=45.91.116.14:4191 , piece_id=5SWPQ2TBXYVYNFOGMH6QVANJVGNFEVOJ6UFMAX74DCC2WWKRSBNA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.215826575 (14h19m30.730929141s) [noise=true , address=77.90.31.10:4517 , piece_id=4XNCSILTEXFXI3T5YJLDOHWW2766ZY2YM723VBVF7XYPPTZWBKEA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.215911546 (14h19m30.731014112s) [noise=true , address=77.90.31.16:4733 , piece_id=R6AKPWEKFLEI7NP4XAMYL72SHIKNIORWP4JXFZXLVRTVIOXWGF2Q, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.216710632 (14h19m30.731813201s) [noise=true , address=77.90.31.18:4807 , piece_id=B3PSWCRJNGDMD6LDDM436N3X75NAA4DEEPLDKL2GTBBJBG3I6ZSQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.217564442 (14h19m30.732667019s) [noise=true , address=77.90.31.15:4697 , piece_id=7MTL64JADCQRGS6UHDTT2ZGJFJV26TP4VYEPVM43MFLNXPPHFRQA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.218100336 (14h19m30.733202913s) [noise=true , address=77.90.31.17:4772 , piece_id=76IYY7NQGZ4WZTPRZ5MW7B7ABMNNQJGHKGZHPKLCX4KHAWNJ7XGA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.220843127 (14h19m30.735945696s) [noise=true , address=104.36.82.11:4115 , piece_id=BKAK43Z32DE3HOXYPSW4BH5LUWMQ3TINDTPESBBXOBHA6NDDCQSA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.221894363 (14h19m30.736996944s) [noise=true , address=104.36.82.6:4875 , piece_id=RHXGB245P3CNRWQJPKBZ4RL6BV5AGIHV35TVLDCWIUVCVU2TRZJA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.227829104 (14h19m30.742931668s) [noise=true , address=45.91.116.8:4048 , piece_id=DGULAYZHOXDN4IO3EHSUHYN3Z3GOMWRPSF3IF762Z7PQNDBWWUCA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.326406182 (14h19m30.841508752s) [noise=true , address=104.36.82.9:4005 , piece_id=BJR6LEV57IPYIKASKNYGCMIKM7XMPZ6J3ASRB4B2ZZHAD3WQB47Q, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.326453057 (14h19m30.841555634s) [seg_pos={0 0}, upload=14734]: Segment reached optimal threshold of 13 pieces. Aug 7 23:58:56.326683605 (14h19m30.841786189s) [noise=true , address=104.36.82.2:4731 , piece_id=374ZMGN47ZFQDXM2NI52ZLYMNYNUMB5HD7UZWKFGEKMDDAG4DUAA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Aug 7 23:58:56.326786377 (14h19m30.84188896s) [noise=true , address=104.36.82.5:4840 , piece_id=JKJE7FIIKVDBL2MQF7IWVNMYH63QV72QXJOYARECFS5FJMSAOK4A, seg_pos={0 0}, upload=14734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: write tcp 159.223.143.39:60496->104.36.82.5:4840: use of closed network connection; write tcp 159.223.143.39:60496->104.36.82.5:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60496->104.36.82.5:4840: use of closed network connection Aug 7 23:58:56.326888961 (14h19m30.841991543s) [noise=true , address=104.36.82.16:4295 , piece_id=QFJ444VYPVNGDMZ2WO2MBWMUH2U6M5K6TQ66T6SHQJW2CM245BTA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: read tcp 159.223.143.39:34618->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Aug 7 23:58:56.326913735 (14h19m30.842016315s) [seg_pos={0 0}, upload=14734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:58:56.326998778 (14h19m30.842101358s) [upload=14734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:58:56.327004732 (14h19m30.84210731s) [upload=14734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:56.375303093 (14h19m30.89040566s) [upload=14734]: Done uploading segments... Aug 7 23:58:56.375321956 (14h19m30.890424523s) [upload=14734]: Upload finished. err: <nil> Aug 7 23:59:26.492160234 (14h20m1.007262811s) [upload=14742]: Starting upload Aug 7 23:59:26.492250238 (14h20m1.007352804s) [upload=14742]: Done starting upload Aug 7 23:59:26.492323252 (14h20m1.00742583s) [upload=14742]: Uploading segments... Aug 7 23:59:26.492605785 (14h20m1.007708349s) [upload=14742]: Got next segment. Inline: false Aug 7 23:59:26.492681402 (14h20m1.007783987s) [upload=14742]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:26.492689520 (14h20m1.007792101s) [upload=14742]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:59:56.459090933 (14h20m30.974193512s) [upload=14742]: Next segment err: context deadline exceeded Aug 7 23:59:56.459165231 (14h20m30.974267796s) [upload=14742]: Done uploading segments... Aug 7 23:59:56.459181138 (14h20m30.974283703s) [upload=14742]: Upload finished. err: context deadline exceeded Aug 7 23:59:56.572495425 (14h20m31.087597994s) [upload=14751]: Starting upload Aug 7 23:59:56.572578962 (14h20m31.087681544s) [upload=14751]: Done starting upload Aug 7 23:59:56.572818459 (14h20m31.087921029s) [upload=14751]: Uploading segments... Aug 7 23:59:56.572953386 (14h20m31.088055953s) [upload=14751]: Got next segment. Inline: false Aug 7 23:59:56.573048904 (14h20m31.088151483s) [upload=14751]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:56.573066041 (14h20m31.088168618s) [upload=14751]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:00:26.529103664 (14h21m1.044206252s) [upload=14751]: Next segment err: context deadline exceeded Aug 8 00:00:26.529241226 (14h21m1.044343811s) [upload=14751]: Done uploading segments... Aug 8 00:00:26.529265086 (14h21m1.044367654s) [upload=14751]: Upload finished. err: context deadline exceeded Duration: 557.71708ms core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 36.005936ms target: https://us1.storj.io/login FAIL (2025-08-08 00:08:08 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 36.03293ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [X...X.X......................................X.XXX...X......] duration: 997.819132ms FAIL (2025-08-08 00:08:00 UTC) operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 474260 (14h23m36.87657683s) [upload=14802]: Got next segment. Inline: false Aug 8 00:03:02.361540620 (14h23m36.876643199s) [upload=14802]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:03:02.361546434 (14h23m36.87664901s) [upload=14802]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:03:57.203920880 (14h24m31.719023446s) [seg_pos={0 0}, upload=14802]: Begin upload segment... Aug 8 00:03:57.203988004 (14h24m31.719090583s) [seg_pos={0 0}, upload=14802]: Done begin upload segment. Aug 8 00:03:57.204020830 (14h24m31.719123417s) [noise=true , address=45.91.116.10:4107 , piece_id=XM3BQK4U7EPQ4VIWN6I44B5U25TDUXWEK2TEFNT7FDNCBKU6DTWA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204041644 (14h24m31.71914423s) [noise=true , address=104.36.82.23:4449 , piece_id=TAFC4ZPBW4MY6RC5TX4CAUPBCJMHXORTGIXDID23ICWC43R6NPWQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204477526 (14h24m31.719580107s) [noise=true , address=79.127.201.215:4346 , piece_id=GMA63JIOTMY7O66ASAO35CP25HFTLKYXVT3HHIJE3GGJJ2WZT4RA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204519100 (14h24m31.719621687s) [noise=true , address=104.36.82.9:4022 , piece_id=N7RQZECLABTXW3N6O4K6CY245IGHUUPGSHLNIENEGDXA2KZUNHGQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204895465 (14h24m31.719998052s) [noise=true , address=104.36.82.38:4700 , piece_id=BERTXTXRJUHPIWKAIPEG4GYJ355BDN6K7UFF3AEYW2YMGQWFEJKQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204925306 (14h24m31.720027871s) [noise=true , address=64.49.13.27:4960 , piece_id=LAJD7GW4V4WEKS5EMUZ2XSL4T2VEL74LBL2PXQF2HE5TFZM2W5JQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.205328523 (14h24m31.720431105s) [noise=true , address=104.36.82.47:4545 , piece_id=QZTD52366FHJIE36E2BLHPZO2Q2EUONODVNHY3NKXLYMNPFTP5TQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.205769139 (14h24m31.720871721s) [noise=true , address=104.36.82.5:4854 , piece_id=C55AYUS7YLDCAWRE6EUWGEAJFBPME2N5IFK4PUJJT6XCFFVCD7OQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.207436252 (14h24m31.722538839s) [noise=true , address=45.91.116.6:4008 , piece_id=OMLZN2SLPT4SNYRDIUTCVMOERA4ALP6YP4QS4AVIG66VRWQNNGNQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.207828410 (14h24m31.722930995s) [noise=true , address=104.36.82.36:4629 , piece_id=3SZDIUDWSAZAWWG4BHVGGAQPU7Q7XWWIQWOKTROV6NPOWUALK5BA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.208468343 (14h24m31.723570931s) [noise=true , address=77.90.31.14:4684 , piece_id=EVAAF4LC6E3JVN6AOBWPCXTP4ILJSQDJP47ZFJHAYPGEO5KEA5MQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.209378344 (14h24m31.72448093s) [noise=true , address=64.49.13.154:4927 , piece_id=GUE6TO7SOONGAQ3UWXEAMDVBN2PDRKYTNADRTH6FWPIPKVEX5JOQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.210191405 (14h24m31.72529399s) [noise=true , address=104.36.82.43:4853 , piece_id=M5GL6IYJ2XTIJRCPWYBISDDYC33OUNTY6RRAKWP374PJQT33AVTQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.211289253 (14h24m31.726391837s) [noise=true , address=104.36.82.46:4509 , piece_id=IYIMDZG6RGFTWDNZYHSZ6FNGN6CGVXNF6VQNXSDUG3US7VSNN7UA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.211675380 (14h24m31.726777962s) [noise=true , address=104.36.82.3:4794 , piece_id=6URKBCOCEX2K3ZROKDT5GI24ZP642SB7WYCU4IYOPQ2XWMHHXMTQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.212135467 (14h24m31.727238053s) [noise=true , address=193.149.164.143:4695 , piece_id=3QVTVIGKIEY4QZWZOUDAQWSKROHOKNR6XTG4DXXUWIVJMNWQEIZA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.212506215 (14h24m31.727608796s) [noise=true , address=104.36.82.34:4565 , piece_id=Q5YZ7G72VW7QBR64RLS2HMQX5CU3L5AELKGZ5Q3J74EPOGSBGNWA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.212830381 (14h24m31.727932969s) [noise=true , address=104.36.82.62:4421 , piece_id=XSTGRY73EHC33KBNGDTYUZOSLBQS5Z3ZY6QOPAEDER4NTRPG3B3Q, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.294294492 (14h24m31.809397074s) [upload=14802]: Next returned nil segment Aug 8 00:03:57.294311485 (14h24m31.809414071s) [upload=14802]: Waiting for error group managing segments... Aug 8 00:03:57.306352633 (14h24m31.821455213s) [noise=true , address=104.36.82.34:4565 , piece_id=Q5YZ7G72VW7QBR64RLS2HMQX5CU3L5AELKGZ5Q3J74EPOGSBGNWA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.307346446 (14h24m31.822449009s) [noise=true , address=45.91.116.6:4008 , piece_id=OMLZN2SLPT4SNYRDIUTCVMOERA4ALP6YP4QS4AVIG66VRWQNNGNQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.308700400 (14h24m31.823802978s) [noise=true , address=104.36.82.36:4629 , piece_id=3SZDIUDWSAZAWWG4BHVGGAQPU7Q7XWWIQWOKTROV6NPOWUALK5BA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.312160131 (14h24m31.827262707s) [noise=true , address=104.36.82.62:4421 , piece_id=XSTGRY73EHC33KBNGDTYUZOSLBQS5Z3ZY6QOPAEDER4NTRPG3B3Q, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.320102036 (14h24m31.835204602s) [noise=true , address=104.36.82.47:4545 , piece_id=QZTD52366FHJIE36E2BLHPZO2Q2EUONODVNHY3NKXLYMNPFTP5TQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.321164433 (14h24m31.836267014s) [noise=true , address=45.91.116.10:4107 , piece_id=XM3BQK4U7EPQ4VIWN6I44B5U25TDUXWEK2TEFNT7FDNCBKU6DTWA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.326337358 (14h24m31.841439935s) [noise=true , address=77.90.31.14:4684 , piece_id=EVAAF4LC6E3JVN6AOBWPCXTP4ILJSQDJP47ZFJHAYPGEO5KEA5MQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.352532224 (14h24m31.867634786s) [noise=true , address=104.36.82.23:4449 , piece_id=TAFC4ZPBW4MY6RC5TX4CAUPBCJMHXORTGIXDID23ICWC43R6NPWQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.356261323 (14h24m31.871363886s) [noise=true , address=104.36.82.43:4853 , piece_id=M5GL6IYJ2XTIJRCPWYBISDDYC33OUNTY6RRAKWP374PJQT33AVTQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.423088525 (14h24m31.938191096s) [noise=true , address=104.36.82.38:4700 , piece_id=BERTXTXRJUHPIWKAIPEG4GYJ355BDN6K7UFF3AEYW2YMGQWFEJKQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.471455369 (14h24m31.986557939s) [noise=true , address=79.127.201.215:4346 , piece_id=GMA63JIOTMY7O66ASAO35CP25HFTLKYXVT3HHIJE3GGJJ2WZT4RA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.749384271 (14h24m32.264486838s) [noise=true , address=104.36.82.9:4022 , piece_id=N7RQZECLABTXW3N6O4K6CY245IGHUUPGSHLNIENEGDXA2KZUNHGQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.873513358 (14h24m32.388615936s) [noise=true , address=193.149.164.143:4695 , piece_id=3QVTVIGKIEY4QZWZOUDAQWSKROHOKNR6XTG4DXXUWIVJMNWQEIZA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.876598294 (14h24m32.391700872s) [noise=true , address=64.49.13.27:4960 , piece_id=LAJD7GW4V4WEKS5EMUZ2XSL4T2VEL74LBL2PXQF2HE5TFZM2W5JQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.876652272 (14h24m32.391754851s) [seg_pos={0 0}, upload=14802]: Segment reached optimal threshold of 14 pieces. Aug 8 00:03:57.877023576 (14h24m32.392126154s) [noise=true , address=104.36.82.5:4854 , piece_id=C55AYUS7YLDCAWRE6EUWGEAJFBPME2N5IFK4PUJJT6XCFFVCD7OQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19pxQXokdTJ9W6k5aZrzGZSoAn9p1NqpwbTWMSC6U7ncr8ipwy): protocol: write tcp 159.223.143.39:37798->104.36.82.5:4854: use of closed network connection; write tcp 159.223.143.39:37798->104.36.82.5:4854: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37798->104.36.82.5:4854: use of closed network connection Aug 8 00:03:57.877160473 (14h24m32.392263043s) [noise=true , address=64.49.13.154:4927 , piece_id=GUE6TO7SOONGAQ3UWXEAMDVBN2PDRKYTNADRTH6FWPIPKVEX5JOQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12U8eKLnBHxvuutyABVGSgUJbG3kr9Cce4sDx59LQiJHT1W2U8k): protocol: expected piece hash; context canceled; EOF Aug 8 00:03:57.877284905 (14h24m32.392387475s) [noise=true , address=104.36.82.46:4509 , piece_id=IYIMDZG6RGFTWDNZYHSZ6FNGN6CGVXNF6VQNXSDUG3US7VSNN7UA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y2NmauNvCyrMJ5z66Lihb3h3C9nwdJYqfSfNmC5dwDGYQqUKh): protocol: expected piece hash; context canceled; EOF Aug 8 00:03:57.877449504 (14h24m32.392552073s) [noise=true , address=104.36.82.3:4794 , piece_id=6URKBCOCEX2K3ZROKDT5GI24ZP642SB7WYCU4IYOPQ2XWMHHXMTQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ctLLD5skk4t5i3BFYgoTmuZ2aqsX23wDGDMnKriQ7U2KXyPAo): protocol: write tcp 159.223.143.39:47368->104.36.82.3:4794: use of closed network connection; context canceled Aug 8 00:03:57.877469652 (14h24m32.392572221s) [seg_pos={0 0}, upload=14802]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:03:57.877571490 (14h24m32.392674067s) [upload=14802]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:03:57.877576879 (14h24m32.392679458s) [upload=14802]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:04:02.023423762 (14h24m36.538526336s) [upload=14802]: Done uploading segments... Aug 8 00:04:02.023447427 (14h24m36.538550004s) [upload=14802]: Upload finished. err: <nil> Duration: 997.842248ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 8 out of 60 is failed (threshold=3) [..........................................XXXXXXX.X.........] duration: 453.951007ms FAIL (2025-08-08 00:08:00 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ne starting upload Aug 7 23:52:04.519569621 (14h12m39.034672186s) [upload=14619]: Uploading segments... Aug 7 23:52:04.519675504 (14h12m39.034778091s) [upload=14619]: Got next segment. Inline: true Aug 7 23:52:04.519701072 (14h12m39.034803649s) [upload=14619]: Waiting for error group managing segments... Aug 7 23:52:04.519755555 (14h12m39.034858131s) [upload=14619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:04.519760650 (14h12m39.034863226s) [upload=14619]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:04.519777078 (14h12m39.034879654s) [upload=14619]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:04.588422656 (14h12m39.103525241s) [upload=14619]: Done uploading segments... Aug 7 23:52:04.588444719 (14h12m39.103547297s) [upload=14619]: Upload finished. err: <nil> Aug 7 23:53:08.147087231 (14h13m42.662189804s) [upload=14641]: Starting upload Aug 7 23:53:08.147181411 (14h13m42.662283996s) [upload=14641]: Done starting upload Aug 7 23:53:08.147281202 (14h13m42.66238377s) [upload=14641]: Uploading segments... Aug 7 23:53:08.147322830 (14h13m42.662425415s) [upload=14641]: Got next segment. Inline: true Aug 7 23:53:08.147327701 (14h13m42.662430277s) [upload=14641]: Waiting for error group managing segments... Aug 7 23:53:08.147455761 (14h13m42.66255833s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:08.147460709 (14h13m42.662563285s) [upload=14641]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:08.147463890 (14h13m42.662566457s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:08.116969456 (14h14m42.632072039s) [upload=14641]: Done uploading segments... Aug 7 23:54:08.117007036 (14h14m42.632109618s) [upload=14641]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:22.831162336 (14h14m57.346264913s) [upload=14661]: Starting upload Aug 7 23:54:22.831234431 (14h14m57.346337008s) [upload=14661]: Done starting upload Aug 7 23:54:22.831306794 (14h14m57.346409362s) [upload=14661]: Uploading segments... Aug 7 23:54:22.831343257 (14h14m57.346445842s) [upload=14661]: Got next segment. Inline: true Aug 7 23:54:22.831347808 (14h14m57.346450375s) [upload=14661]: Waiting for error group managing segments... Aug 7 23:54:22.831399492 (14h14m57.346502071s) [upload=14661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:22.831404349 (14h14m57.346506915s) [upload=14661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:22.831407696 (14h14m57.346510265s) [upload=14661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:22.799123829 (14h15m57.314226409s) [upload=14661]: Done uploading segments... Aug 7 23:55:22.799154191 (14h15m57.314256772s) [upload=14661]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:22.918485444 (14h15m57.433588022s) [upload=14677]: Starting upload Aug 7 23:55:22.918566621 (14h15m57.433672141s) [upload=14677]: Done starting upload Aug 7 23:55:22.918647605 (14h15m57.433750172s) [upload=14677]: Uploading segments... Aug 7 23:55:22.918705315 (14h15m57.4338079s) [upload=14677]: Got next segment. Inline: true Aug 7 23:55:22.918710403 (14h15m57.43381298s) [upload=14677]: Waiting for error group managing segments... Aug 7 23:55:22.918764176 (14h15m57.433866753s) [upload=14677]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:22.918769239 (14h15m57.433871815s) [upload=14677]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:22.918772352 (14h15m57.433874928s) [upload=14677]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:22.891103920 (14h16m57.406206502s) [upload=14677]: Done uploading segments... Aug 7 23:56:22.891142892 (14h16m57.406245471s) [upload=14677]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:23.012512156 (14h16m57.527614733s) [upload=14690]: Starting upload Aug 7 23:56:23.012570683 (14h16m57.527673265s) [upload=14690]: Done starting upload Aug 7 23:56:23.012632890 (14h16m57.527735455s) [upload=14690]: Uploading segments... Aug 7 23:56:23.012675514 (14h16m57.527778093s) [upload=14690]: Got next segment. Inline: true Aug 7 23:56:23.012680516 (14h16m57.527783092s) [upload=14690]: Waiting for error group managing segments... Aug 7 23:56:23.012734720 (14h16m57.527837295s) [upload=14690]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:23.012739356 (14h16m57.527841923s) [upload=14690]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:23.012742484 (14h16m57.527845059s) [upload=14690]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:22.981126119 (14h17m57.496228686s) [upload=14690]: Done uploading segments... Aug 7 23:57:22.981169317 (14h17m57.496271889s) [upload=14690]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:23.078169780 (14h17m57.593272348s) [upload=14705]: Starting upload Aug 7 23:57:23.078287503 (14h17m57.593390079s) [upload=14705]: Done starting upload Aug 7 23:57:23.078353530 (14h17m57.593456107s) [upload=14705]: Uploading segments... Aug 7 23:57:23.078394403 (14h17m57.593496992s) [upload=14705]: Got next segment. Inline: true Aug 7 23:57:23.078400888 (14h17m57.593503457s) [upload=14705]: Waiting for error group managing segments... Aug 7 23:57:23.078452630 (14h17m57.593555201s) [upload=14705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:23.078457270 (14h17m57.593559838s) [upload=14705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:23.078460483 (14h17m57.593563048s) [upload=14705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:23.049396934 (14h18m57.564499518s) [upload=14705]: Done uploading segments... Aug 7 23:58:23.049418059 (14h18m57.564520634s) [upload=14705]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:23.169720285 (14h18m57.684822852s) [upload=14722]: Starting upload Aug 7 23:58:23.169779212 (14h18m57.684881788s) [upload=14722]: Done starting upload Aug 7 23:58:23.169840834 (14h18m57.684943401s) [upload=14722]: Uploading segments... Aug 7 23:58:23.169880437 (14h18m57.684983021s) [upload=14722]: Got next segment. Inline: true Aug 7 23:58:23.169885772 (14h18m57.684988338s) [upload=14722]: Waiting for error group managing segments... Aug 7 23:58:23.169927728 (14h18m57.685030303s) [upload=14722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:23.169933919 (14h18m57.685036498s) [upload=14722]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:23.169938117 (14h18m57.685040704s) [upload=14722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:23.138140642 (14h19m57.653243209s) [upload=14722]: Done uploading segments... Aug 7 23:59:23.138159469 (14h19m57.653262035s) [upload=14722]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:35.502179618 (14h20m10.017282184s) [upload=14745]: Starting upload Aug 7 23:59:35.502246881 (14h20m10.017349449s) [upload=14745]: Done starting upload Aug 7 23:59:35.502308502 (14h20m10.017411071s) [upload=14745]: Uploading segments... Aug 7 23:59:35.502347394 (14h20m10.017449979s) [upload=14745]: Got next segment. Inline: true Aug 7 23:59:35.502351750 (14h20m10.017454327s) [upload=14745]: Waiting for error group managing segments... Aug 7 23:59:35.502390443 (14h20m10.017493009s) [upload=14745]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:35.502394821 (14h20m10.017497398s) [upload=14745]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:35.502398119 (14h20m10.017500695s) [upload=14745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:35.472447782 (14h21m9.987550364s) [upload=14745]: Done uploading segments... Aug 8 00:00:35.472473966 (14h21m9.987576558s) [upload=14745]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:00:35.593243192 (14h21m10.10834576s) [upload=14760]: Starting upload Aug 8 00:00:35.593395835 (14h21m10.108498415s) [upload=14760]: Done starting upload Aug 8 00:00:35.593497926 (14h21m10.10860049s) [upload=14760]: Uploading segments... Aug 8 00:00:35.593659481 (14h21m10.108762045s) [upload=14760]: Got next segment. Inline: true Aug 8 00:00:35.593667686 (14h21m10.108770273s) [upload=14760]: Waiting for error group managing segments... Aug 8 00:00:35.593750345 (14h21m10.10885293s) [upload=14760]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:35.593773930 (14h21m10.108876508s) [upload=14760]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:35.593782409 (14h21m10.108884991s) [upload=14760]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:35.662793141 (14h21m10.177895709s) [upload=14760]: Done uploading segments... Aug 8 00:00:35.662815087 (14h21m10.177917658s) [upload=14760]: Upload finished. err: <nil> Aug 8 00:01:23.648690375 (14h21m58.163792959s) [upload=14775]: Starting upload Aug 8 00:01:23.648799550 (14h21m58.163902126s) [upload=14775]: Done starting upload Aug 8 00:01:23.648928369 (14h21m58.164030934s) [upload=14775]: Uploading segments... Aug 8 00:01:23.648986999 (14h21m58.164089578s) [upload=14775]: Got next segment. Inline: true Aug 8 00:01:23.648992059 (14h21m58.164094634s) [upload=14775]: Waiting for error group managing segments... Aug 8 00:01:23.649037148 (14h21m58.164139714s) [upload=14775]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:23.649041722 (14h21m58.164144298s) [upload=14775]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:23.649044950 (14h21m58.164147527s) [upload=14775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:23.618855090 (14h22m58.133957671s) [upload=14775]: Done uploading segments... Aug 8 00:02:23.618900564 (14h22m58.134003142s) [upload=14775]: Upload finished. err: metaclient: context deadline exceeded Duration: 454.001699ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X..........X...........] duration: 731.648057ms FAIL (2025-08-08 00:08:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 731.671058ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 54.064683ms FAIL (2025-08-08 00:07:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== piece... Aug 8 00:06:47.146014029 (14h27m21.661116615s) [noise=true , address=104.36.82.23:4456 , piece_id=6NTCQI462IGFCUSV6ZV2QVB2JB6XSIMBH54ABTWGOFPVNK74ON5Q, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.146042952 (14h27m21.661145531s) [noise=true , address=79.127.205.233:4636 , piece_id=XSBDMHKCPCIIQOV7SMEHSU5USOTUTVLNE2K4VHTI2O44IV5EGQBQ, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.146392229 (14h27m21.661494798s) [noise=true , address=104.36.82.9:4010 , piece_id=3MZMFKQ2LIPOSVDHEMGJHF5O3K7US3LKYSEGXPTKZMWDYMDKAOSQ, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.146489907 (14h27m21.661592493s) [noise=true , address=104.36.82.44:4883 , piece_id=CIX3VQVGNNZ2SSQYPXNZA3QZPQILYDXWGB66D3XMV3XZFG5IJDAA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.146751379 (14h27m21.661853965s) [noise=true , address=64.49.13.137:4494 , piece_id=XHXS6IICALFTEEQ6L3ARHBJMAQ6Z7AFIUIWWKSLCASRZVSY5IM2A, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.146970578 (14h27m21.662073166s) [noise=true , address=104.36.82.43:4868 , piece_id=7ZBUMTM2KGHZVQGM44SPYHZIK6BYKZ4RJLQNGEOLXK5FIBDOQQJA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.147102314 (14h27m21.662204899s) [noise=true , address=45.91.116.8:4065 , piece_id=FRBY35WXD6GF4WVNQXC5U4BDFGAKMCACN3GYCQYM3KIU7VR5F7EQ, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.147440038 (14h27m21.662542625s) [noise=true , address=104.36.82.13:4195 , piece_id=SE3RX356M46XAAHYGXSM56S4KC4QVSR4S3WPHHMLJHL2Y2U5RLOQ, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.147464022 (14h27m21.662566606s) [noise=true , address=64.49.13.27:4968 , piece_id=CG322ANVVFEA6XYPOW3FAJMWAHM3GJ72LCWWJMLMR6WZP2JIYM7A, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.147806500 (14h27m21.662909085s) [noise=true , address=104.36.82.14:4231 , piece_id=MDMME3XQYMMMTY2D2AO4JB2SHPJ6A4UPWN7DWXVPRI4EUAALNNBA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.148072099 (14h27m21.663174685s) [noise=true , address=104.36.82.60:4346 , piece_id=6ZKPSHS3NT77FZPZSN2BPX5A2T62JQJRJKD3H25QVZEPB34YDY3A, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.148231380 (14h27m21.663333959s) [noise=true , address=77.90.31.11:4557 , piece_id=VX6K65NE4IXJGBJEAC6GWXVMVFMOWJJJKUKWZSSG6XNCNFHH4IHA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.148503333 (14h27m21.663605919s) [noise=true , address=104.36.82.54:4163 , piece_id=5BYDGKBFV6DAPJOBSY52IWQ5HHLF6JPZ7PN6R7TY26KVL6CGI46Q, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.151364728 (14h27m21.666467312s) [noise=true , address=104.36.82.22:4511 , piece_id=DH5JH2TMIJEM7QXT3DIFIYQOOXAJNRUZAQD6QP7ZHUBZARBIXASA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.151909137 (14h27m21.667011732s) [noise=true , address=104.36.82.47:4527 , piece_id=LMI25BZRKDKTM442UA3DBFAFOI45RNSBXNS6K3J63GZKNFCIKLMQ, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.152324438 (14h27m21.667427024s) [noise=true , address=104.36.82.62:4416 , piece_id=C3BGVDY6FCK2WXISID4FQYE2XR3Q2SW5N5DETAUNVEZAFOT7MYXA, seg_pos={0 0}, upload=14868]: Uploading piece... Aug 8 00:06:47.269249030 (14h27m21.7843516s) [upload=14868]: Next returned nil segment Aug 8 00:06:47.269265635 (14h27m21.784368211s) [upload=14868]: Waiting for error group managing segments... Aug 8 00:06:47.280030107 (14h27m21.795132688s) [noise=true , address=104.36.82.62:4416 , piece_id=C3BGVDY6FCK2WXISID4FQYE2XR3Q2SW5N5DETAUNVEZAFOT7MYXA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.282243133 (14h27m21.797345711s) [noise=true , address=104.36.82.43:4868 , piece_id=7ZBUMTM2KGHZVQGM44SPYHZIK6BYKZ4RJLQNGEOLXK5FIBDOQQJA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.282935406 (14h27m21.798037984s) [noise=true , address=104.36.82.9:4010 , piece_id=3MZMFKQ2LIPOSVDHEMGJHF5O3K7US3LKYSEGXPTKZMWDYMDKAOSQ, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.283797129 (14h27m21.798899704s) [noise=true , address=104.36.82.13:4195 , piece_id=SE3RX356M46XAAHYGXSM56S4KC4QVSR4S3WPHHMLJHL2Y2U5RLOQ, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.288848212 (14h27m21.803950789s) [noise=true , address=45.91.116.5:4301 , piece_id=GQLYE5Z2Q2SHC2WIEZMTPRGYQ7B4EQZZYJLVC4NQMRENIDZATA5Q, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.292885927 (14h27m21.807988496s) [noise=true , address=45.91.116.8:4065 , piece_id=FRBY35WXD6GF4WVNQXC5U4BDFGAKMCACN3GYCQYM3KIU7VR5F7EQ, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.296224440 (14h27m21.811327016s) [noise=true , address=104.36.82.14:4231 , piece_id=MDMME3XQYMMMTY2D2AO4JB2SHPJ6A4UPWN7DWXVPRI4EUAALNNBA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.299834425 (14h27m21.81493699s) [noise=true , address=104.36.82.54:4163 , piece_id=5BYDGKBFV6DAPJOBSY52IWQ5HHLF6JPZ7PN6R7TY26KVL6CGI46Q, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.305883863 (14h27m21.820986439s) [noise=true , address=104.36.82.23:4456 , piece_id=6NTCQI462IGFCUSV6ZV2QVB2JB6XSIMBH54ABTWGOFPVNK74ON5Q, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.309497417 (14h27m21.824599987s) [noise=true , address=104.36.82.47:4527 , piece_id=LMI25BZRKDKTM442UA3DBFAFOI45RNSBXNS6K3J63GZKNFCIKLMQ, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.323758805 (14h27m21.838861383s) [noise=true , address=77.90.31.11:4557 , piece_id=VX6K65NE4IXJGBJEAC6GWXVMVFMOWJJJKUKWZSSG6XNCNFHH4IHA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.326822007 (14h27m21.841924593s) [noise=true , address=104.36.82.45:4938 , piece_id=QI275JMAE4DVWGO5CY3IDTC6WKZCNPPCO3LKE5EK74TGJSBLB2VA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.327668872 (14h27m21.842771449s) [noise=true , address=104.36.82.22:4511 , piece_id=DH5JH2TMIJEM7QXT3DIFIYQOOXAJNRUZAQD6QP7ZHUBZARBIXASA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.339100720 (14h27m21.854203297s) [noise=true , address=104.36.82.44:4883 , piece_id=CIX3VQVGNNZ2SSQYPXNZA3QZPQILYDXWGB66D3XMV3XZFG5IJDAA, seg_pos={0 0}, upload=14868]: Done uploading piece. err: <nil> Aug 8 00:06:47.339134259 (14h27m21.854236839s) [seg_pos={0 0}, upload=14868]: Segment reached optimal threshold of 14 pieces. Aug 8 00:06:47.339343619 (14h27m21.854446181s) [noise=true , address=104.36.82.60:4346 , piece_id=6ZKPSHS3NT77FZPZSN2BPX5A2T62JQJRJKD3H25QVZEPB34YDY3A, seg_pos={0 0}, upload=14868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HohFM5Npi28Xq65JVMGUDK2bfX7xh5uYXNeHs3XW2VRH8rS72): protocol: expected piece hash; context canceled; EOF Aug 8 00:06:47.339560288 (14h27m21.854662868s) [noise=true , address=79.127.205.233:4636 , piece_id=XSBDMHKCPCIIQOV7SMEHSU5USOTUTVLNE2K4VHTI2O44IV5EGQBQ, seg_pos={0 0}, upload=14868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mTJmjiHwMJqSkuWKrFnAgJUt1gjQ4wPjfgY7qouWcanaNUy46): protocol: read tcp 159.223.143.39:35278->79.127.205.233:4636: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:06:47.339701256 (14h27m21.854803837s) [noise=true , address=64.49.13.137:4494 , piece_id=XHXS6IICALFTEEQ6L3ARHBJMAQ6Z7AFIUIWWKSLCASRZVSY5IM2A, seg_pos={0 0}, upload=14868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TX66fDVv3qu4yBGx9FNZTiArNeUnqxjMYwb3VGqYCQRe8RwoU): protocol: read tcp 159.223.143.39:57032->64.49.13.137:4494: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 8 00:06:47.339837792 (14h27m21.854940379s) [noise=true , address=64.49.13.27:4968 , piece_id=CG322ANVVFEA6XYPOW3FAJMWAHM3GJ72LCWWJMLMR6WZP2JIYM7A, seg_pos={0 0}, upload=14868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iMqzmqAR9V6Zxnd3x9pw6HVmcg5UaELNM9nTgGkYiYDxeo1Qk): protocol: read tcp 159.223.143.39:48324->64.49.13.27:4968: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Aug 8 00:06:47.339968534 (14h27m21.855071118s) [seg_pos={0 0}, upload=14868]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:06:47.340079739 (14h27m21.855182316s) [upload=14868]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:06:47.340086599 (14h27m21.855189182s) [upload=14868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:06:47.400977058 (14h27m21.916079641s) [upload=14868]: Done uploading segments... Aug 8 00:06:47.401023653 (14h27m21.916126232s) [upload=14868]: Upload finished. err: <nil> Aug 8 00:07:57.606754320 (14h28m32.121856897s) [upload=14881]: Starting upload Aug 8 00:07:57.606822750 (14h28m32.121925318s) [upload=14881]: Done starting upload Aug 8 00:07:57.607097187 (14h28m32.122199756s) [upload=14881]: Uploading segments... Aug 8 00:07:57.607354625 (14h28m32.122457207s) [upload=14881]: Got next segment. Inline: false Aug 8 00:07:57.607435490 (14h28m32.122538073s) [upload=14881]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:07:57.607441733 (14h28m32.122544313s) [upload=14881]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:07:57.622249420 (14h28m32.137352005s) [upload=14881]: Next segment err: metaclient: Too Many Requests Aug 8 00:07:57.622307892 (14h28m32.137410469s) [upload=14881]: Done uploading segments... Aug 8 00:07:57.622322946 (14h28m32.137425515s) [upload=14881]: Upload finished. err: metaclient: Too Many Requests Duration: 54.091772ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 6 out of 60 is failed (threshold=3) [........................................X.XX.X.....X..X.....] duration: 264.952899ms FAIL (2025-08-08 00:07:54 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Upload finished. err: <nil> Aug 7 23:56:22.913855555 (14h16m57.428958117s) [upload=14689]: Starting upload Aug 7 23:56:22.913925154 (14h16m57.429027734s) [upload=14689]: Done starting upload Aug 7 23:56:22.913999531 (14h16m57.429102093s) [upload=14689]: Uploading segments... Aug 7 23:56:22.914048699 (14h16m57.429151282s) [upload=14689]: Got next segment. Inline: true Aug 7 23:56:22.914052623 (14h16m57.4291552s) [upload=14689]: Waiting for error group managing segments... Aug 7 23:56:22.914089928 (14h16m57.429192503s) [upload=14689]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:22.914093434 (14h16m57.42919601s) [upload=14689]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:22.914096175 (14h16m57.429198751s) [upload=14689]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:52.884393262 (14h17m27.399495844s) [upload=14689]: Done uploading segments... Aug 7 23:56:52.884415558 (14h17m27.399518126s) [upload=14689]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:09.822697015 (14h17m44.337799585s) [upload=14701]: Starting upload Aug 7 23:57:09.822774891 (14h17m44.337877456s) [upload=14701]: Done starting upload Aug 7 23:57:09.822876390 (14h17m44.337978955s) [upload=14701]: Uploading segments... Aug 7 23:57:09.823345112 (14h17m44.338447694s) [upload=14701]: Got next segment. Inline: true Aug 7 23:57:09.823357163 (14h17m44.338459742s) [upload=14701]: Waiting for error group managing segments... Aug 7 23:57:09.823423761 (14h17m44.338526328s) [upload=14701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:09.823429578 (14h17m44.338532154s) [upload=14701]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:09.823433695 (14h17m44.338536271s) [upload=14701]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:09.886245350 (14h17m44.401347936s) [upload=14701]: Done uploading segments... Aug 7 23:57:09.886269151 (14h17m44.401371729s) [upload=14701]: Upload finished. err: <nil> Aug 7 23:58:11.646122622 (14h18m46.161225204s) [upload=14719]: Starting upload Aug 7 23:58:11.646203494 (14h18m46.16130607s) [upload=14719]: Done starting upload Aug 7 23:58:11.646278294 (14h18m46.161380873s) [upload=14719]: Uploading segments... Aug 7 23:58:11.646339731 (14h18m46.161442316s) [upload=14719]: Got next segment. Inline: true Aug 7 23:58:11.646344984 (14h18m46.16144756s) [upload=14719]: Waiting for error group managing segments... Aug 7 23:58:11.646388852 (14h18m46.161491429s) [upload=14719]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:11.646393482 (14h18m46.161496048s) [upload=14719]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:11.646396765 (14h18m46.161499342s) [upload=14719]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:11.705007157 (14h18m46.220109734s) [upload=14719]: Done uploading segments... Aug 7 23:58:11.705026176 (14h18m46.220128742s) [upload=14719]: Upload finished. err: <nil> Aug 7 23:58:41.852871585 (14h19m16.367974172s) [upload=14731]: Starting upload Aug 7 23:58:41.852974809 (14h19m16.368077394s) [upload=14731]: Done starting upload Aug 7 23:58:41.853095753 (14h19m16.368198318s) [upload=14731]: Uploading segments... Aug 7 23:58:41.853172118 (14h19m16.368274707s) [upload=14731]: Got next segment. Inline: true Aug 7 23:58:41.853178991 (14h19m16.368281577s) [upload=14731]: Waiting for error group managing segments... Aug 7 23:58:41.853256548 (14h19m16.368359135s) [upload=14731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:41.853263687 (14h19m16.368366273s) [upload=14731]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:41.853268996 (14h19m16.368371576s) [upload=14731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:41.919489485 (14h19m16.434592068s) [upload=14731]: Done uploading segments... Aug 7 23:58:41.919512553 (14h19m16.434615126s) [upload=14731]: Upload finished. err: <nil> Aug 7 23:59:44.072708375 (14h20m18.587810945s) [upload=14749]: Starting upload Aug 7 23:59:44.072763052 (14h20m18.587865629s) [upload=14749]: Done starting upload Aug 7 23:59:44.072818447 (14h20m18.587921015s) [upload=14749]: Uploading segments... Aug 7 23:59:44.072859036 (14h20m18.587961622s) [upload=14749]: Got next segment. Inline: true Aug 7 23:59:44.072863633 (14h20m18.587966209s) [upload=14749]: Waiting for error group managing segments... Aug 7 23:59:44.072902482 (14h20m18.588005058s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:44.072906670 (14h20m18.588009237s) [upload=14749]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:44.072910115 (14h20m18.588012692s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:44.143691065 (14h20m18.658793888s) [upload=14749]: Done uploading segments... Aug 7 23:59:44.143716427 (14h20m18.658819013s) [upload=14749]: Upload finished. err: <nil> Aug 8 00:00:34.719721220 (14h21m9.234823804s) [upload=14759]: Starting upload Aug 8 00:00:34.719816520 (14h21m9.2349191s) [upload=14759]: Done starting upload Aug 8 00:00:34.719975322 (14h21m9.235077899s) [upload=14759]: Uploading segments... Aug 8 00:00:34.720108487 (14h21m9.235211073s) [upload=14759]: Got next segment. Inline: true Aug 8 00:00:34.720116703 (14h21m9.235219288s) [upload=14759]: Waiting for error group managing segments... Aug 8 00:00:34.720194402 (14h21m9.235296985s) [upload=14759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:34.720201518 (14h21m9.235304101s) [upload=14759]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:34.720205160 (14h21m9.23530774s) [upload=14759]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:34.794300564 (14h21m9.309403157s) [upload=14759]: Done uploading segments... Aug 8 00:00:34.794338844 (14h21m9.309441422s) [upload=14759]: Upload finished. err: <nil> Aug 8 00:01:01.776429007 (14h21m36.291531596s) [upload=14768]: Starting upload Aug 8 00:01:01.776506683 (14h21m36.291609246s) [upload=14768]: Done starting upload Aug 8 00:01:01.776613033 (14h21m36.291715611s) [upload=14768]: Uploading segments... Aug 8 00:01:01.776661337 (14h21m36.291763928s) [upload=14768]: Got next segment. Inline: true Aug 8 00:01:01.776666749 (14h21m36.29176933s) [upload=14768]: Waiting for error group managing segments... Aug 8 00:01:01.776715825 (14h21m36.291818408s) [upload=14768]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:01.776720751 (14h21m36.291823332s) [upload=14768]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:01.776724804 (14h21m36.291827383s) [upload=14768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:31.728405977 (14h22m6.243508555s) [upload=14768]: Done uploading segments... Aug 8 00:01:31.728424052 (14h22m6.243526628s) [upload=14768]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:02:03.679985597 (14h22m38.19508816s) [upload=14786]: Starting upload Aug 8 00:02:03.680087933 (14h22m38.19519052s) [upload=14786]: Done starting upload Aug 8 00:02:03.680194558 (14h22m38.195297135s) [upload=14786]: Uploading segments... Aug 8 00:02:03.680249278 (14h22m38.195351863s) [upload=14786]: Got next segment. Inline: true Aug 8 00:02:03.680254458 (14h22m38.195357034s) [upload=14786]: Waiting for error group managing segments... Aug 8 00:02:03.680309917 (14h22m38.195412483s) [upload=14786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:03.680314576 (14h22m38.195417152s) [upload=14786]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:03.680318116 (14h22m38.195420692s) [upload=14786]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:03.773554102 (14h22m38.288656668s) [upload=14786]: Done uploading segments... Aug 8 00:02:03.773587490 (14h22m38.288690069s) [upload=14786]: Upload finished. err: <nil> Aug 8 00:02:50.218217757 (14h23m24.733320413s) [upload=14799]: Starting upload Aug 8 00:02:50.218299459 (14h23m24.733402029s) [upload=14799]: Done starting upload Aug 8 00:02:50.218366696 (14h23m24.733469274s) [upload=14799]: Uploading segments... Aug 8 00:02:50.218404998 (14h23m24.733507583s) [upload=14799]: Got next segment. Inline: true Aug 8 00:02:50.218409405 (14h23m24.733511982s) [upload=14799]: Waiting for error group managing segments... Aug 8 00:02:50.218452564 (14h23m24.733555139s) [upload=14799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:50.218457853 (14h23m24.733560433s) [upload=14799]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:50.218462652 (14h23m24.733565229s) [upload=14799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:50.299378321 (14h23m24.814480907s) [upload=14799]: Done uploading segments... Aug 8 00:02:50.299417555 (14h23m24.814520132s) [upload=14799]: Upload finished. err: <nil> Aug 8 00:03:43.640902837 (14h24m18.156005417s) [upload=14815]: Starting upload Aug 8 00:03:43.640984215 (14h24m18.1560868s) [upload=14815]: Done starting upload Aug 8 00:03:43.641134510 (14h24m18.156237076s) [upload=14815]: Uploading segments... Aug 8 00:03:43.641193969 (14h24m18.156296557s) [upload=14815]: Got next segment. Inline: true Aug 8 00:03:43.641200481 (14h24m18.156303062s) [upload=14815]: Waiting for error group managing segments... Aug 8 00:03:43.641360742 (14h24m18.156463325s) [upload=14815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:03:43.641376086 (14h24m18.156478665s) [upload=14815]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:03:43.641379649 (14h24m18.156482228s) [upload=14815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:04:13.608341576 (14h24m48.123444144s) [upload=14815]: Done uploading segments... Aug 8 00:04:13.608361002 (14h24m48.123463569s) [upload=14815]: Upload finished. err: metaclient: context deadline exceeded Duration: 264.979024ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [.....................................X..X....XX...X.........] duration: 280.498964ms FAIL (2025-08-08 00:07:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 280.518286ms edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 32.712633ms target: https://us1.storj.io/login FAIL (2025-08-08 00:07:44 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 32.741039ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [............................................XX..XX..........] duration: 1.044041452s FAIL (2025-08-08 00:07:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.04407862s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [......................................X..XXX..X...X.........] duration: 244.557712ms FAIL (2025-08-08 00:07:23 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 244.602914ms core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [......................................X..X....XX...X........] duration: 268.734178ms FAIL (2025-08-08 00:07:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 268.775257ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [XXXX.............................................X..........] duration: 2.094119421s FAIL (2025-08-08 00:07:10 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.094149861s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.....................................X..........X..........] duration: 516.67274ms FAIL (2025-08-08 00:07:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 516.695137ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [XX...X.X......................................X.XXX...X.....] duration: 1.224935679s FAIL (2025-08-08 00:07:02 UTC) operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 474260 (14h23m36.87657683s) [upload=14802]: Got next segment. Inline: false Aug 8 00:03:02.361540620 (14h23m36.876643199s) [upload=14802]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:03:02.361546434 (14h23m36.87664901s) [upload=14802]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:03:57.203920880 (14h24m31.719023446s) [seg_pos={0 0}, upload=14802]: Begin upload segment... Aug 8 00:03:57.203988004 (14h24m31.719090583s) [seg_pos={0 0}, upload=14802]: Done begin upload segment. Aug 8 00:03:57.204020830 (14h24m31.719123417s) [noise=true , address=45.91.116.10:4107 , piece_id=XM3BQK4U7EPQ4VIWN6I44B5U25TDUXWEK2TEFNT7FDNCBKU6DTWA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204041644 (14h24m31.71914423s) [noise=true , address=104.36.82.23:4449 , piece_id=TAFC4ZPBW4MY6RC5TX4CAUPBCJMHXORTGIXDID23ICWC43R6NPWQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204477526 (14h24m31.719580107s) [noise=true , address=79.127.201.215:4346 , piece_id=GMA63JIOTMY7O66ASAO35CP25HFTLKYXVT3HHIJE3GGJJ2WZT4RA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204519100 (14h24m31.719621687s) [noise=true , address=104.36.82.9:4022 , piece_id=N7RQZECLABTXW3N6O4K6CY245IGHUUPGSHLNIENEGDXA2KZUNHGQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204895465 (14h24m31.719998052s) [noise=true , address=104.36.82.38:4700 , piece_id=BERTXTXRJUHPIWKAIPEG4GYJ355BDN6K7UFF3AEYW2YMGQWFEJKQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204925306 (14h24m31.720027871s) [noise=true , address=64.49.13.27:4960 , piece_id=LAJD7GW4V4WEKS5EMUZ2XSL4T2VEL74LBL2PXQF2HE5TFZM2W5JQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.205328523 (14h24m31.720431105s) [noise=true , address=104.36.82.47:4545 , piece_id=QZTD52366FHJIE36E2BLHPZO2Q2EUONODVNHY3NKXLYMNPFTP5TQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.205769139 (14h24m31.720871721s) [noise=true , address=104.36.82.5:4854 , piece_id=C55AYUS7YLDCAWRE6EUWGEAJFBPME2N5IFK4PUJJT6XCFFVCD7OQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.207436252 (14h24m31.722538839s) [noise=true , address=45.91.116.6:4008 , piece_id=OMLZN2SLPT4SNYRDIUTCVMOERA4ALP6YP4QS4AVIG66VRWQNNGNQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.207828410 (14h24m31.722930995s) [noise=true , address=104.36.82.36:4629 , piece_id=3SZDIUDWSAZAWWG4BHVGGAQPU7Q7XWWIQWOKTROV6NPOWUALK5BA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.208468343 (14h24m31.723570931s) [noise=true , address=77.90.31.14:4684 , piece_id=EVAAF4LC6E3JVN6AOBWPCXTP4ILJSQDJP47ZFJHAYPGEO5KEA5MQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.209378344 (14h24m31.72448093s) [noise=true , address=64.49.13.154:4927 , piece_id=GUE6TO7SOONGAQ3UWXEAMDVBN2PDRKYTNADRTH6FWPIPKVEX5JOQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.210191405 (14h24m31.72529399s) [noise=true , address=104.36.82.43:4853 , piece_id=M5GL6IYJ2XTIJRCPWYBISDDYC33OUNTY6RRAKWP374PJQT33AVTQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.211289253 (14h24m31.726391837s) [noise=true , address=104.36.82.46:4509 , piece_id=IYIMDZG6RGFTWDNZYHSZ6FNGN6CGVXNF6VQNXSDUG3US7VSNN7UA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.211675380 (14h24m31.726777962s) [noise=true , address=104.36.82.3:4794 , piece_id=6URKBCOCEX2K3ZROKDT5GI24ZP642SB7WYCU4IYOPQ2XWMHHXMTQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.212135467 (14h24m31.727238053s) [noise=true , address=193.149.164.143:4695 , piece_id=3QVTVIGKIEY4QZWZOUDAQWSKROHOKNR6XTG4DXXUWIVJMNWQEIZA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.212506215 (14h24m31.727608796s) [noise=true , address=104.36.82.34:4565 , piece_id=Q5YZ7G72VW7QBR64RLS2HMQX5CU3L5AELKGZ5Q3J74EPOGSBGNWA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.212830381 (14h24m31.727932969s) [noise=true , address=104.36.82.62:4421 , piece_id=XSTGRY73EHC33KBNGDTYUZOSLBQS5Z3ZY6QOPAEDER4NTRPG3B3Q, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.294294492 (14h24m31.809397074s) [upload=14802]: Next returned nil segment Aug 8 00:03:57.294311485 (14h24m31.809414071s) [upload=14802]: Waiting for error group managing segments... Aug 8 00:03:57.306352633 (14h24m31.821455213s) [noise=true , address=104.36.82.34:4565 , piece_id=Q5YZ7G72VW7QBR64RLS2HMQX5CU3L5AELKGZ5Q3J74EPOGSBGNWA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.307346446 (14h24m31.822449009s) [noise=true , address=45.91.116.6:4008 , piece_id=OMLZN2SLPT4SNYRDIUTCVMOERA4ALP6YP4QS4AVIG66VRWQNNGNQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.308700400 (14h24m31.823802978s) [noise=true , address=104.36.82.36:4629 , piece_id=3SZDIUDWSAZAWWG4BHVGGAQPU7Q7XWWIQWOKTROV6NPOWUALK5BA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.312160131 (14h24m31.827262707s) [noise=true , address=104.36.82.62:4421 , piece_id=XSTGRY73EHC33KBNGDTYUZOSLBQS5Z3ZY6QOPAEDER4NTRPG3B3Q, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.320102036 (14h24m31.835204602s) [noise=true , address=104.36.82.47:4545 , piece_id=QZTD52366FHJIE36E2BLHPZO2Q2EUONODVNHY3NKXLYMNPFTP5TQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.321164433 (14h24m31.836267014s) [noise=true , address=45.91.116.10:4107 , piece_id=XM3BQK4U7EPQ4VIWN6I44B5U25TDUXWEK2TEFNT7FDNCBKU6DTWA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.326337358 (14h24m31.841439935s) [noise=true , address=77.90.31.14:4684 , piece_id=EVAAF4LC6E3JVN6AOBWPCXTP4ILJSQDJP47ZFJHAYPGEO5KEA5MQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.352532224 (14h24m31.867634786s) [noise=true , address=104.36.82.23:4449 , piece_id=TAFC4ZPBW4MY6RC5TX4CAUPBCJMHXORTGIXDID23ICWC43R6NPWQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.356261323 (14h24m31.871363886s) [noise=true , address=104.36.82.43:4853 , piece_id=M5GL6IYJ2XTIJRCPWYBISDDYC33OUNTY6RRAKWP374PJQT33AVTQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.423088525 (14h24m31.938191096s) [noise=true , address=104.36.82.38:4700 , piece_id=BERTXTXRJUHPIWKAIPEG4GYJ355BDN6K7UFF3AEYW2YMGQWFEJKQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.471455369 (14h24m31.986557939s) [noise=true , address=79.127.201.215:4346 , piece_id=GMA63JIOTMY7O66ASAO35CP25HFTLKYXVT3HHIJE3GGJJ2WZT4RA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.749384271 (14h24m32.264486838s) [noise=true , address=104.36.82.9:4022 , piece_id=N7RQZECLABTXW3N6O4K6CY245IGHUUPGSHLNIENEGDXA2KZUNHGQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.873513358 (14h24m32.388615936s) [noise=true , address=193.149.164.143:4695 , piece_id=3QVTVIGKIEY4QZWZOUDAQWSKROHOKNR6XTG4DXXUWIVJMNWQEIZA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.876598294 (14h24m32.391700872s) [noise=true , address=64.49.13.27:4960 , piece_id=LAJD7GW4V4WEKS5EMUZ2XSL4T2VEL74LBL2PXQF2HE5TFZM2W5JQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.876652272 (14h24m32.391754851s) [seg_pos={0 0}, upload=14802]: Segment reached optimal threshold of 14 pieces. Aug 8 00:03:57.877023576 (14h24m32.392126154s) [noise=true , address=104.36.82.5:4854 , piece_id=C55AYUS7YLDCAWRE6EUWGEAJFBPME2N5IFK4PUJJT6XCFFVCD7OQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19pxQXokdTJ9W6k5aZrzGZSoAn9p1NqpwbTWMSC6U7ncr8ipwy): protocol: write tcp 159.223.143.39:37798->104.36.82.5:4854: use of closed network connection; write tcp 159.223.143.39:37798->104.36.82.5:4854: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37798->104.36.82.5:4854: use of closed network connection Aug 8 00:03:57.877160473 (14h24m32.392263043s) [noise=true , address=64.49.13.154:4927 , piece_id=GUE6TO7SOONGAQ3UWXEAMDVBN2PDRKYTNADRTH6FWPIPKVEX5JOQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12U8eKLnBHxvuutyABVGSgUJbG3kr9Cce4sDx59LQiJHT1W2U8k): protocol: expected piece hash; context canceled; EOF Aug 8 00:03:57.877284905 (14h24m32.392387475s) [noise=true , address=104.36.82.46:4509 , piece_id=IYIMDZG6RGFTWDNZYHSZ6FNGN6CGVXNF6VQNXSDUG3US7VSNN7UA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y2NmauNvCyrMJ5z66Lihb3h3C9nwdJYqfSfNmC5dwDGYQqUKh): protocol: expected piece hash; context canceled; EOF Aug 8 00:03:57.877449504 (14h24m32.392552073s) [noise=true , address=104.36.82.3:4794 , piece_id=6URKBCOCEX2K3ZROKDT5GI24ZP642SB7WYCU4IYOPQ2XWMHHXMTQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ctLLD5skk4t5i3BFYgoTmuZ2aqsX23wDGDMnKriQ7U2KXyPAo): protocol: write tcp 159.223.143.39:47368->104.36.82.3:4794: use of closed network connection; context canceled Aug 8 00:03:57.877469652 (14h24m32.392572221s) [seg_pos={0 0}, upload=14802]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:03:57.877571490 (14h24m32.392674067s) [upload=14802]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:03:57.877576879 (14h24m32.392679458s) [upload=14802]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:04:02.023423762 (14h24m36.538526336s) [upload=14802]: Done uploading segments... Aug 8 00:04:02.023447427 (14h24m36.538550004s) [upload=14802]: Upload finished. err: <nil> Duration: 1.224967486s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......................................XXX.X...XX.XX........] duration: 478.194001ms FAIL (2025-08-08 00:07:01 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 0.31.10:4517 , piece_id=4XNCSILTEXFXI3T5YJLDOHWW2766ZY2YM723VBVF7XYPPTZWBKEA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.130393931 (14h19m30.645496508s) [noise=true , address=45.91.116.8:4048 , piece_id=DGULAYZHOXDN4IO3EHSUHYN3Z3GOMWRPSF3IF762Z7PQNDBWWUCA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.130898433 (14h19m30.646001001s) [noise=true , address=104.36.82.11:4115 , piece_id=BKAK43Z32DE3HOXYPSW4BH5LUWMQ3TINDTPESBBXOBHA6NDDCQSA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.131654285 (14h19m30.646756854s) [noise=true , address=77.90.31.16:4733 , piece_id=R6AKPWEKFLEI7NP4XAMYL72SHIKNIORWP4JXFZXLVRTVIOXWGF2Q, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.132299167 (14h19m30.647401729s) [noise=true , address=45.91.116.13:4153 , piece_id=G35ZWVBI2OZYXYXOY5RKDG6IO2G4PET4BDFHGLQV4NVTUVAGVNJQ, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.132815896 (14h19m30.647918459s) [noise=true , address=77.90.31.17:4772 , piece_id=76IYY7NQGZ4WZTPRZ5MW7B7ABMNNQJGHKGZHPKLCX4KHAWNJ7XGA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.133305230 (14h19m30.648407796s) [noise=true , address=104.36.82.6:4875 , piece_id=RHXGB245P3CNRWQJPKBZ4RL6BV5AGIHV35TVLDCWIUVCVU2TRZJA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.134064921 (14h19m30.649167503s) [noise=true , address=104.36.82.2:4731 , piece_id=374ZMGN47ZFQDXM2NI52ZLYMNYNUMB5HD7UZWKFGEKMDDAG4DUAA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.134501761 (14h19m30.649604335s) [noise=true , address=104.36.82.5:4840 , piece_id=JKJE7FIIKVDBL2MQF7IWVNMYH63QV72QXJOYARECFS5FJMSAOK4A, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.135039838 (14h19m30.650142405s) [noise=true , address=45.91.116.14:4191 , piece_id=5SWPQ2TBXYVYNFOGMH6QVANJVGNFEVOJ6UFMAX74DCC2WWKRSBNA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.135450095 (14h19m30.650552665s) [noise=true , address=77.90.31.19:4842 , piece_id=OUTUWUFXDDTLL52UJ6RTLORFEPWMQVL6JYBP3PPFY2DHS7JQHMBQ, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.135750292 (14h19m30.650852861s) [noise=true , address=77.90.31.15:4697 , piece_id=7MTL64JADCQRGS6UHDTT2ZGJFJV26TP4VYEPVM43MFLNXPPHFRQA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.136062113 (14h19m30.651164675s) [noise=true , address=104.36.82.9:4005 , piece_id=BJR6LEV57IPYIKASKNYGCMIKM7XMPZ6J3ASRB4B2ZZHAD3WQB47Q, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.136413662 (14h19m30.65151623s) [noise=true , address=45.91.116.9:4263 , piece_id=3G4QVYQG6AJ555B632PNRZWUUJM23K3GWSIPGNWJBCQR3ZZGZ4BQ, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.136771535 (14h19m30.651874107s) [noise=true , address=104.36.82.16:4295 , piece_id=QFJ444VYPVNGDMZ2WO2MBWMUH2U6M5K6TQ66T6SHQJW2CM245BTA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.209688234 (14h19m30.724790803s) [upload=14734]: Next returned nil segment Aug 7 23:58:56.209703014 (14h19m30.72480559s) [upload=14734]: Waiting for error group managing segments... Aug 7 23:58:56.214389537 (14h19m30.729492107s) [noise=true , address=45.91.116.9:4263 , piece_id=3G4QVYQG6AJ555B632PNRZWUUJM23K3GWSIPGNWJBCQR3ZZGZ4BQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.214445477 (14h19m30.729548054s) [noise=true , address=45.91.116.13:4153 , piece_id=G35ZWVBI2OZYXYXOY5RKDG6IO2G4PET4BDFHGLQV4NVTUVAGVNJQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.214979544 (14h19m30.73008212s) [noise=true , address=77.90.31.19:4842 , piece_id=OUTUWUFXDDTLL52UJ6RTLORFEPWMQVL6JYBP3PPFY2DHS7JQHMBQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.215082462 (14h19m30.730185028s) [noise=true , address=45.91.116.14:4191 , piece_id=5SWPQ2TBXYVYNFOGMH6QVANJVGNFEVOJ6UFMAX74DCC2WWKRSBNA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.215826575 (14h19m30.730929141s) [noise=true , address=77.90.31.10:4517 , piece_id=4XNCSILTEXFXI3T5YJLDOHWW2766ZY2YM723VBVF7XYPPTZWBKEA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.215911546 (14h19m30.731014112s) [noise=true , address=77.90.31.16:4733 , piece_id=R6AKPWEKFLEI7NP4XAMYL72SHIKNIORWP4JXFZXLVRTVIOXWGF2Q, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.216710632 (14h19m30.731813201s) [noise=true , address=77.90.31.18:4807 , piece_id=B3PSWCRJNGDMD6LDDM436N3X75NAA4DEEPLDKL2GTBBJBG3I6ZSQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.217564442 (14h19m30.732667019s) [noise=true , address=77.90.31.15:4697 , piece_id=7MTL64JADCQRGS6UHDTT2ZGJFJV26TP4VYEPVM43MFLNXPPHFRQA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.218100336 (14h19m30.733202913s) [noise=true , address=77.90.31.17:4772 , piece_id=76IYY7NQGZ4WZTPRZ5MW7B7ABMNNQJGHKGZHPKLCX4KHAWNJ7XGA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.220843127 (14h19m30.735945696s) [noise=true , address=104.36.82.11:4115 , piece_id=BKAK43Z32DE3HOXYPSW4BH5LUWMQ3TINDTPESBBXOBHA6NDDCQSA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.221894363 (14h19m30.736996944s) [noise=true , address=104.36.82.6:4875 , piece_id=RHXGB245P3CNRWQJPKBZ4RL6BV5AGIHV35TVLDCWIUVCVU2TRZJA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.227829104 (14h19m30.742931668s) [noise=true , address=45.91.116.8:4048 , piece_id=DGULAYZHOXDN4IO3EHSUHYN3Z3GOMWRPSF3IF762Z7PQNDBWWUCA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.326406182 (14h19m30.841508752s) [noise=true , address=104.36.82.9:4005 , piece_id=BJR6LEV57IPYIKASKNYGCMIKM7XMPZ6J3ASRB4B2ZZHAD3WQB47Q, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.326453057 (14h19m30.841555634s) [seg_pos={0 0}, upload=14734]: Segment reached optimal threshold of 13 pieces. Aug 7 23:58:56.326683605 (14h19m30.841786189s) [noise=true , address=104.36.82.2:4731 , piece_id=374ZMGN47ZFQDXM2NI52ZLYMNYNUMB5HD7UZWKFGEKMDDAG4DUAA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Aug 7 23:58:56.326786377 (14h19m30.84188896s) [noise=true , address=104.36.82.5:4840 , piece_id=JKJE7FIIKVDBL2MQF7IWVNMYH63QV72QXJOYARECFS5FJMSAOK4A, seg_pos={0 0}, upload=14734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: write tcp 159.223.143.39:60496->104.36.82.5:4840: use of closed network connection; write tcp 159.223.143.39:60496->104.36.82.5:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60496->104.36.82.5:4840: use of closed network connection Aug 7 23:58:56.326888961 (14h19m30.841991543s) [noise=true , address=104.36.82.16:4295 , piece_id=QFJ444VYPVNGDMZ2WO2MBWMUH2U6M5K6TQ66T6SHQJW2CM245BTA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: read tcp 159.223.143.39:34618->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Aug 7 23:58:56.326913735 (14h19m30.842016315s) [seg_pos={0 0}, upload=14734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:58:56.326998778 (14h19m30.842101358s) [upload=14734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:58:56.327004732 (14h19m30.84210731s) [upload=14734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:56.375303093 (14h19m30.89040566s) [upload=14734]: Done uploading segments... Aug 7 23:58:56.375321956 (14h19m30.890424523s) [upload=14734]: Upload finished. err: <nil> Aug 7 23:59:26.492160234 (14h20m1.007262811s) [upload=14742]: Starting upload Aug 7 23:59:26.492250238 (14h20m1.007352804s) [upload=14742]: Done starting upload Aug 7 23:59:26.492323252 (14h20m1.00742583s) [upload=14742]: Uploading segments... Aug 7 23:59:26.492605785 (14h20m1.007708349s) [upload=14742]: Got next segment. Inline: false Aug 7 23:59:26.492681402 (14h20m1.007783987s) [upload=14742]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:26.492689520 (14h20m1.007792101s) [upload=14742]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:59:56.459090933 (14h20m30.974193512s) [upload=14742]: Next segment err: context deadline exceeded Aug 7 23:59:56.459165231 (14h20m30.974267796s) [upload=14742]: Done uploading segments... Aug 7 23:59:56.459181138 (14h20m30.974283703s) [upload=14742]: Upload finished. err: context deadline exceeded Aug 7 23:59:56.572495425 (14h20m31.087597994s) [upload=14751]: Starting upload Aug 7 23:59:56.572578962 (14h20m31.087681544s) [upload=14751]: Done starting upload Aug 7 23:59:56.572818459 (14h20m31.087921029s) [upload=14751]: Uploading segments... Aug 7 23:59:56.572953386 (14h20m31.088055953s) [upload=14751]: Got next segment. Inline: false Aug 7 23:59:56.573048904 (14h20m31.088151483s) [upload=14751]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:56.573066041 (14h20m31.088168618s) [upload=14751]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:00:26.529103664 (14h21m1.044206252s) [upload=14751]: Next segment err: context deadline exceeded Aug 8 00:00:26.529241226 (14h21m1.044343811s) [upload=14751]: Done uploading segments... Aug 8 00:00:26.529265086 (14h21m1.044367654s) [upload=14751]: Upload finished. err: context deadline exceeded Duration: 478.223886ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [.......................................X..XXX..X...X........] duration: 245.191608ms FAIL (2025-08-08 00:06:58 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 245.232181ms core uplink us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 30.238526ms target: https://us1.storj.io/login FAIL (2025-08-08 00:06:58 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 30.276056ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.............................................XX..XX.........] duration: 1.095123769s FAIL (2025-08-08 00:06:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.095183415s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 8 out of 60 is failed (threshold=3) [...........................................XXXXXXX.X........] duration: 389.680443ms FAIL (2025-08-08 00:06:45 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ne starting upload Aug 7 23:52:04.519569621 (14h12m39.034672186s) [upload=14619]: Uploading segments... Aug 7 23:52:04.519675504 (14h12m39.034778091s) [upload=14619]: Got next segment. Inline: true Aug 7 23:52:04.519701072 (14h12m39.034803649s) [upload=14619]: Waiting for error group managing segments... Aug 7 23:52:04.519755555 (14h12m39.034858131s) [upload=14619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:04.519760650 (14h12m39.034863226s) [upload=14619]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:04.519777078 (14h12m39.034879654s) [upload=14619]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:04.588422656 (14h12m39.103525241s) [upload=14619]: Done uploading segments... Aug 7 23:52:04.588444719 (14h12m39.103547297s) [upload=14619]: Upload finished. err: <nil> Aug 7 23:53:08.147087231 (14h13m42.662189804s) [upload=14641]: Starting upload Aug 7 23:53:08.147181411 (14h13m42.662283996s) [upload=14641]: Done starting upload Aug 7 23:53:08.147281202 (14h13m42.66238377s) [upload=14641]: Uploading segments... Aug 7 23:53:08.147322830 (14h13m42.662425415s) [upload=14641]: Got next segment. Inline: true Aug 7 23:53:08.147327701 (14h13m42.662430277s) [upload=14641]: Waiting for error group managing segments... Aug 7 23:53:08.147455761 (14h13m42.66255833s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:08.147460709 (14h13m42.662563285s) [upload=14641]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:08.147463890 (14h13m42.662566457s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:08.116969456 (14h14m42.632072039s) [upload=14641]: Done uploading segments... Aug 7 23:54:08.117007036 (14h14m42.632109618s) [upload=14641]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:22.831162336 (14h14m57.346264913s) [upload=14661]: Starting upload Aug 7 23:54:22.831234431 (14h14m57.346337008s) [upload=14661]: Done starting upload Aug 7 23:54:22.831306794 (14h14m57.346409362s) [upload=14661]: Uploading segments... Aug 7 23:54:22.831343257 (14h14m57.346445842s) [upload=14661]: Got next segment. Inline: true Aug 7 23:54:22.831347808 (14h14m57.346450375s) [upload=14661]: Waiting for error group managing segments... Aug 7 23:54:22.831399492 (14h14m57.346502071s) [upload=14661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:22.831404349 (14h14m57.346506915s) [upload=14661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:22.831407696 (14h14m57.346510265s) [upload=14661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:22.799123829 (14h15m57.314226409s) [upload=14661]: Done uploading segments... Aug 7 23:55:22.799154191 (14h15m57.314256772s) [upload=14661]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:22.918485444 (14h15m57.433588022s) [upload=14677]: Starting upload Aug 7 23:55:22.918566621 (14h15m57.433672141s) [upload=14677]: Done starting upload Aug 7 23:55:22.918647605 (14h15m57.433750172s) [upload=14677]: Uploading segments... Aug 7 23:55:22.918705315 (14h15m57.4338079s) [upload=14677]: Got next segment. Inline: true Aug 7 23:55:22.918710403 (14h15m57.43381298s) [upload=14677]: Waiting for error group managing segments... Aug 7 23:55:22.918764176 (14h15m57.433866753s) [upload=14677]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:22.918769239 (14h15m57.433871815s) [upload=14677]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:22.918772352 (14h15m57.433874928s) [upload=14677]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:22.891103920 (14h16m57.406206502s) [upload=14677]: Done uploading segments... Aug 7 23:56:22.891142892 (14h16m57.406245471s) [upload=14677]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:23.012512156 (14h16m57.527614733s) [upload=14690]: Starting upload Aug 7 23:56:23.012570683 (14h16m57.527673265s) [upload=14690]: Done starting upload Aug 7 23:56:23.012632890 (14h16m57.527735455s) [upload=14690]: Uploading segments... Aug 7 23:56:23.012675514 (14h16m57.527778093s) [upload=14690]: Got next segment. Inline: true Aug 7 23:56:23.012680516 (14h16m57.527783092s) [upload=14690]: Waiting for error group managing segments... Aug 7 23:56:23.012734720 (14h16m57.527837295s) [upload=14690]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:23.012739356 (14h16m57.527841923s) [upload=14690]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:23.012742484 (14h16m57.527845059s) [upload=14690]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:22.981126119 (14h17m57.496228686s) [upload=14690]: Done uploading segments... Aug 7 23:57:22.981169317 (14h17m57.496271889s) [upload=14690]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:23.078169780 (14h17m57.593272348s) [upload=14705]: Starting upload Aug 7 23:57:23.078287503 (14h17m57.593390079s) [upload=14705]: Done starting upload Aug 7 23:57:23.078353530 (14h17m57.593456107s) [upload=14705]: Uploading segments... Aug 7 23:57:23.078394403 (14h17m57.593496992s) [upload=14705]: Got next segment. Inline: true Aug 7 23:57:23.078400888 (14h17m57.593503457s) [upload=14705]: Waiting for error group managing segments... Aug 7 23:57:23.078452630 (14h17m57.593555201s) [upload=14705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:23.078457270 (14h17m57.593559838s) [upload=14705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:23.078460483 (14h17m57.593563048s) [upload=14705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:23.049396934 (14h18m57.564499518s) [upload=14705]: Done uploading segments... Aug 7 23:58:23.049418059 (14h18m57.564520634s) [upload=14705]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:23.169720285 (14h18m57.684822852s) [upload=14722]: Starting upload Aug 7 23:58:23.169779212 (14h18m57.684881788s) [upload=14722]: Done starting upload Aug 7 23:58:23.169840834 (14h18m57.684943401s) [upload=14722]: Uploading segments... Aug 7 23:58:23.169880437 (14h18m57.684983021s) [upload=14722]: Got next segment. Inline: true Aug 7 23:58:23.169885772 (14h18m57.684988338s) [upload=14722]: Waiting for error group managing segments... Aug 7 23:58:23.169927728 (14h18m57.685030303s) [upload=14722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:23.169933919 (14h18m57.685036498s) [upload=14722]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:23.169938117 (14h18m57.685040704s) [upload=14722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:23.138140642 (14h19m57.653243209s) [upload=14722]: Done uploading segments... Aug 7 23:59:23.138159469 (14h19m57.653262035s) [upload=14722]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:35.502179618 (14h20m10.017282184s) [upload=14745]: Starting upload Aug 7 23:59:35.502246881 (14h20m10.017349449s) [upload=14745]: Done starting upload Aug 7 23:59:35.502308502 (14h20m10.017411071s) [upload=14745]: Uploading segments... Aug 7 23:59:35.502347394 (14h20m10.017449979s) [upload=14745]: Got next segment. Inline: true Aug 7 23:59:35.502351750 (14h20m10.017454327s) [upload=14745]: Waiting for error group managing segments... Aug 7 23:59:35.502390443 (14h20m10.017493009s) [upload=14745]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:35.502394821 (14h20m10.017497398s) [upload=14745]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:35.502398119 (14h20m10.017500695s) [upload=14745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:35.472447782 (14h21m9.987550364s) [upload=14745]: Done uploading segments... Aug 8 00:00:35.472473966 (14h21m9.987576558s) [upload=14745]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:00:35.593243192 (14h21m10.10834576s) [upload=14760]: Starting upload Aug 8 00:00:35.593395835 (14h21m10.108498415s) [upload=14760]: Done starting upload Aug 8 00:00:35.593497926 (14h21m10.10860049s) [upload=14760]: Uploading segments... Aug 8 00:00:35.593659481 (14h21m10.108762045s) [upload=14760]: Got next segment. Inline: true Aug 8 00:00:35.593667686 (14h21m10.108770273s) [upload=14760]: Waiting for error group managing segments... Aug 8 00:00:35.593750345 (14h21m10.10885293s) [upload=14760]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:35.593773930 (14h21m10.108876508s) [upload=14760]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:35.593782409 (14h21m10.108884991s) [upload=14760]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:35.662793141 (14h21m10.177895709s) [upload=14760]: Done uploading segments... Aug 8 00:00:35.662815087 (14h21m10.177917658s) [upload=14760]: Upload finished. err: <nil> Aug 8 00:01:23.648690375 (14h21m58.163792959s) [upload=14775]: Starting upload Aug 8 00:01:23.648799550 (14h21m58.163902126s) [upload=14775]: Done starting upload Aug 8 00:01:23.648928369 (14h21m58.164030934s) [upload=14775]: Uploading segments... Aug 8 00:01:23.648986999 (14h21m58.164089578s) [upload=14775]: Got next segment. Inline: true Aug 8 00:01:23.648992059 (14h21m58.164094634s) [upload=14775]: Waiting for error group managing segments... Aug 8 00:01:23.649037148 (14h21m58.164139714s) [upload=14775]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:23.649041722 (14h21m58.164144298s) [upload=14775]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:23.649044950 (14h21m58.164147527s) [upload=14775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:23.618855090 (14h22m58.133957671s) [upload=14775]: Done uploading segments... Aug 8 00:02:23.618900564 (14h22m58.134003142s) [upload=14775]: Upload finished. err: metaclient: context deadline exceeded Duration: 389.715889ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 6 out of 60 is failed (threshold=3) [.........................................X.XX.X.....X..X....] duration: 302.328319ms FAIL (2025-08-08 00:06:42 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Upload finished. err: <nil> Aug 7 23:56:22.913855555 (14h16m57.428958117s) [upload=14689]: Starting upload Aug 7 23:56:22.913925154 (14h16m57.429027734s) [upload=14689]: Done starting upload Aug 7 23:56:22.913999531 (14h16m57.429102093s) [upload=14689]: Uploading segments... Aug 7 23:56:22.914048699 (14h16m57.429151282s) [upload=14689]: Got next segment. Inline: true Aug 7 23:56:22.914052623 (14h16m57.4291552s) [upload=14689]: Waiting for error group managing segments... Aug 7 23:56:22.914089928 (14h16m57.429192503s) [upload=14689]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:22.914093434 (14h16m57.42919601s) [upload=14689]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:22.914096175 (14h16m57.429198751s) [upload=14689]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:52.884393262 (14h17m27.399495844s) [upload=14689]: Done uploading segments... Aug 7 23:56:52.884415558 (14h17m27.399518126s) [upload=14689]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:09.822697015 (14h17m44.337799585s) [upload=14701]: Starting upload Aug 7 23:57:09.822774891 (14h17m44.337877456s) [upload=14701]: Done starting upload Aug 7 23:57:09.822876390 (14h17m44.337978955s) [upload=14701]: Uploading segments... Aug 7 23:57:09.823345112 (14h17m44.338447694s) [upload=14701]: Got next segment. Inline: true Aug 7 23:57:09.823357163 (14h17m44.338459742s) [upload=14701]: Waiting for error group managing segments... Aug 7 23:57:09.823423761 (14h17m44.338526328s) [upload=14701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:09.823429578 (14h17m44.338532154s) [upload=14701]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:09.823433695 (14h17m44.338536271s) [upload=14701]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:09.886245350 (14h17m44.401347936s) [upload=14701]: Done uploading segments... Aug 7 23:57:09.886269151 (14h17m44.401371729s) [upload=14701]: Upload finished. err: <nil> Aug 7 23:58:11.646122622 (14h18m46.161225204s) [upload=14719]: Starting upload Aug 7 23:58:11.646203494 (14h18m46.16130607s) [upload=14719]: Done starting upload Aug 7 23:58:11.646278294 (14h18m46.161380873s) [upload=14719]: Uploading segments... Aug 7 23:58:11.646339731 (14h18m46.161442316s) [upload=14719]: Got next segment. Inline: true Aug 7 23:58:11.646344984 (14h18m46.16144756s) [upload=14719]: Waiting for error group managing segments... Aug 7 23:58:11.646388852 (14h18m46.161491429s) [upload=14719]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:11.646393482 (14h18m46.161496048s) [upload=14719]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:11.646396765 (14h18m46.161499342s) [upload=14719]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:11.705007157 (14h18m46.220109734s) [upload=14719]: Done uploading segments... Aug 7 23:58:11.705026176 (14h18m46.220128742s) [upload=14719]: Upload finished. err: <nil> Aug 7 23:58:41.852871585 (14h19m16.367974172s) [upload=14731]: Starting upload Aug 7 23:58:41.852974809 (14h19m16.368077394s) [upload=14731]: Done starting upload Aug 7 23:58:41.853095753 (14h19m16.368198318s) [upload=14731]: Uploading segments... Aug 7 23:58:41.853172118 (14h19m16.368274707s) [upload=14731]: Got next segment. Inline: true Aug 7 23:58:41.853178991 (14h19m16.368281577s) [upload=14731]: Waiting for error group managing segments... Aug 7 23:58:41.853256548 (14h19m16.368359135s) [upload=14731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:41.853263687 (14h19m16.368366273s) [upload=14731]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:41.853268996 (14h19m16.368371576s) [upload=14731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:41.919489485 (14h19m16.434592068s) [upload=14731]: Done uploading segments... Aug 7 23:58:41.919512553 (14h19m16.434615126s) [upload=14731]: Upload finished. err: <nil> Aug 7 23:59:44.072708375 (14h20m18.587810945s) [upload=14749]: Starting upload Aug 7 23:59:44.072763052 (14h20m18.587865629s) [upload=14749]: Done starting upload Aug 7 23:59:44.072818447 (14h20m18.587921015s) [upload=14749]: Uploading segments... Aug 7 23:59:44.072859036 (14h20m18.587961622s) [upload=14749]: Got next segment. Inline: true Aug 7 23:59:44.072863633 (14h20m18.587966209s) [upload=14749]: Waiting for error group managing segments... Aug 7 23:59:44.072902482 (14h20m18.588005058s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:44.072906670 (14h20m18.588009237s) [upload=14749]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:44.072910115 (14h20m18.588012692s) [upload=14749]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:44.143691065 (14h20m18.658793888s) [upload=14749]: Done uploading segments... Aug 7 23:59:44.143716427 (14h20m18.658819013s) [upload=14749]: Upload finished. err: <nil> Aug 8 00:00:34.719721220 (14h21m9.234823804s) [upload=14759]: Starting upload Aug 8 00:00:34.719816520 (14h21m9.2349191s) [upload=14759]: Done starting upload Aug 8 00:00:34.719975322 (14h21m9.235077899s) [upload=14759]: Uploading segments... Aug 8 00:00:34.720108487 (14h21m9.235211073s) [upload=14759]: Got next segment. Inline: true Aug 8 00:00:34.720116703 (14h21m9.235219288s) [upload=14759]: Waiting for error group managing segments... Aug 8 00:00:34.720194402 (14h21m9.235296985s) [upload=14759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:34.720201518 (14h21m9.235304101s) [upload=14759]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:34.720205160 (14h21m9.23530774s) [upload=14759]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:34.794300564 (14h21m9.309403157s) [upload=14759]: Done uploading segments... Aug 8 00:00:34.794338844 (14h21m9.309441422s) [upload=14759]: Upload finished. err: <nil> Aug 8 00:01:01.776429007 (14h21m36.291531596s) [upload=14768]: Starting upload Aug 8 00:01:01.776506683 (14h21m36.291609246s) [upload=14768]: Done starting upload Aug 8 00:01:01.776613033 (14h21m36.291715611s) [upload=14768]: Uploading segments... Aug 8 00:01:01.776661337 (14h21m36.291763928s) [upload=14768]: Got next segment. Inline: true Aug 8 00:01:01.776666749 (14h21m36.29176933s) [upload=14768]: Waiting for error group managing segments... Aug 8 00:01:01.776715825 (14h21m36.291818408s) [upload=14768]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:01.776720751 (14h21m36.291823332s) [upload=14768]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:01.776724804 (14h21m36.291827383s) [upload=14768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:01:31.728405977 (14h22m6.243508555s) [upload=14768]: Done uploading segments... Aug 8 00:01:31.728424052 (14h22m6.243526628s) [upload=14768]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:02:03.679985597 (14h22m38.19508816s) [upload=14786]: Starting upload Aug 8 00:02:03.680087933 (14h22m38.19519052s) [upload=14786]: Done starting upload Aug 8 00:02:03.680194558 (14h22m38.195297135s) [upload=14786]: Uploading segments... Aug 8 00:02:03.680249278 (14h22m38.195351863s) [upload=14786]: Got next segment. Inline: true Aug 8 00:02:03.680254458 (14h22m38.195357034s) [upload=14786]: Waiting for error group managing segments... Aug 8 00:02:03.680309917 (14h22m38.195412483s) [upload=14786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:03.680314576 (14h22m38.195417152s) [upload=14786]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:03.680318116 (14h22m38.195420692s) [upload=14786]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:03.773554102 (14h22m38.288656668s) [upload=14786]: Done uploading segments... Aug 8 00:02:03.773587490 (14h22m38.288690069s) [upload=14786]: Upload finished. err: <nil> Aug 8 00:02:50.218217757 (14h23m24.733320413s) [upload=14799]: Starting upload Aug 8 00:02:50.218299459 (14h23m24.733402029s) [upload=14799]: Done starting upload Aug 8 00:02:50.218366696 (14h23m24.733469274s) [upload=14799]: Uploading segments... Aug 8 00:02:50.218404998 (14h23m24.733507583s) [upload=14799]: Got next segment. Inline: true Aug 8 00:02:50.218409405 (14h23m24.733511982s) [upload=14799]: Waiting for error group managing segments... Aug 8 00:02:50.218452564 (14h23m24.733555139s) [upload=14799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:02:50.218457853 (14h23m24.733560433s) [upload=14799]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:02:50.218462652 (14h23m24.733565229s) [upload=14799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:50.299378321 (14h23m24.814480907s) [upload=14799]: Done uploading segments... Aug 8 00:02:50.299417555 (14h23m24.814520132s) [upload=14799]: Upload finished. err: <nil> Aug 8 00:03:43.640902837 (14h24m18.156005417s) [upload=14815]: Starting upload Aug 8 00:03:43.640984215 (14h24m18.1560868s) [upload=14815]: Done starting upload Aug 8 00:03:43.641134510 (14h24m18.156237076s) [upload=14815]: Uploading segments... Aug 8 00:03:43.641193969 (14h24m18.156296557s) [upload=14815]: Got next segment. Inline: true Aug 8 00:03:43.641200481 (14h24m18.156303062s) [upload=14815]: Waiting for error group managing segments... Aug 8 00:03:43.641360742 (14h24m18.156463325s) [upload=14815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:03:43.641376086 (14h24m18.156478665s) [upload=14815]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:03:43.641379649 (14h24m18.156482228s) [upload=14815]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:04:13.608341576 (14h24m48.123444144s) [upload=14815]: Done uploading segments... Aug 8 00:04:13.608361002 (14h24m48.123463569s) [upload=14815]: Upload finished. err: metaclient: context deadline exceeded Duration: 302.525772ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [.......................................X..X....XX...X.......] duration: 487.118235ms FAIL (2025-08-08 00:06:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 487.363873ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.....................................X..........X.........] duration: 716.057585ms FAIL (2025-08-08 00:06:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 716.094904ms edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 29.623944ms target: https://us1.storj.io/login FAIL (2025-08-08 00:06:26 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 29.654091ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..............................................XX..XX........] duration: 921.025216ms FAIL (2025-08-08 00:06:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 921.051349ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 6 out of 60 is failed (threshold=3) [XXXXX.............................................X.........] duration: 1.824065438s FAIL (2025-08-08 00:06:20 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.824097167s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 8 out of 60 is failed (threshold=3) [............................................XXXXXXX.X.......] duration: 243.272707ms FAIL (2025-08-08 00:06:17 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ne starting upload Aug 7 23:52:04.519569621 (14h12m39.034672186s) [upload=14619]: Uploading segments... Aug 7 23:52:04.519675504 (14h12m39.034778091s) [upload=14619]: Got next segment. Inline: true Aug 7 23:52:04.519701072 (14h12m39.034803649s) [upload=14619]: Waiting for error group managing segments... Aug 7 23:52:04.519755555 (14h12m39.034858131s) [upload=14619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:04.519760650 (14h12m39.034863226s) [upload=14619]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:04.519777078 (14h12m39.034879654s) [upload=14619]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:04.588422656 (14h12m39.103525241s) [upload=14619]: Done uploading segments... Aug 7 23:52:04.588444719 (14h12m39.103547297s) [upload=14619]: Upload finished. err: <nil> Aug 7 23:53:08.147087231 (14h13m42.662189804s) [upload=14641]: Starting upload Aug 7 23:53:08.147181411 (14h13m42.662283996s) [upload=14641]: Done starting upload Aug 7 23:53:08.147281202 (14h13m42.66238377s) [upload=14641]: Uploading segments... Aug 7 23:53:08.147322830 (14h13m42.662425415s) [upload=14641]: Got next segment. Inline: true Aug 7 23:53:08.147327701 (14h13m42.662430277s) [upload=14641]: Waiting for error group managing segments... Aug 7 23:53:08.147455761 (14h13m42.66255833s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:08.147460709 (14h13m42.662563285s) [upload=14641]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:08.147463890 (14h13m42.662566457s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:08.116969456 (14h14m42.632072039s) [upload=14641]: Done uploading segments... Aug 7 23:54:08.117007036 (14h14m42.632109618s) [upload=14641]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:22.831162336 (14h14m57.346264913s) [upload=14661]: Starting upload Aug 7 23:54:22.831234431 (14h14m57.346337008s) [upload=14661]: Done starting upload Aug 7 23:54:22.831306794 (14h14m57.346409362s) [upload=14661]: Uploading segments... Aug 7 23:54:22.831343257 (14h14m57.346445842s) [upload=14661]: Got next segment. Inline: true Aug 7 23:54:22.831347808 (14h14m57.346450375s) [upload=14661]: Waiting for error group managing segments... Aug 7 23:54:22.831399492 (14h14m57.346502071s) [upload=14661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:22.831404349 (14h14m57.346506915s) [upload=14661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:22.831407696 (14h14m57.346510265s) [upload=14661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:22.799123829 (14h15m57.314226409s) [upload=14661]: Done uploading segments... Aug 7 23:55:22.799154191 (14h15m57.314256772s) [upload=14661]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:22.918485444 (14h15m57.433588022s) [upload=14677]: Starting upload Aug 7 23:55:22.918566621 (14h15m57.433672141s) [upload=14677]: Done starting upload Aug 7 23:55:22.918647605 (14h15m57.433750172s) [upload=14677]: Uploading segments... Aug 7 23:55:22.918705315 (14h15m57.4338079s) [upload=14677]: Got next segment. Inline: true Aug 7 23:55:22.918710403 (14h15m57.43381298s) [upload=14677]: Waiting for error group managing segments... Aug 7 23:55:22.918764176 (14h15m57.433866753s) [upload=14677]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:22.918769239 (14h15m57.433871815s) [upload=14677]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:22.918772352 (14h15m57.433874928s) [upload=14677]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:22.891103920 (14h16m57.406206502s) [upload=14677]: Done uploading segments... Aug 7 23:56:22.891142892 (14h16m57.406245471s) [upload=14677]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:23.012512156 (14h16m57.527614733s) [upload=14690]: Starting upload Aug 7 23:56:23.012570683 (14h16m57.527673265s) [upload=14690]: Done starting upload Aug 7 23:56:23.012632890 (14h16m57.527735455s) [upload=14690]: Uploading segments... Aug 7 23:56:23.012675514 (14h16m57.527778093s) [upload=14690]: Got next segment. Inline: true Aug 7 23:56:23.012680516 (14h16m57.527783092s) [upload=14690]: Waiting for error group managing segments... Aug 7 23:56:23.012734720 (14h16m57.527837295s) [upload=14690]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:23.012739356 (14h16m57.527841923s) [upload=14690]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:23.012742484 (14h16m57.527845059s) [upload=14690]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:22.981126119 (14h17m57.496228686s) [upload=14690]: Done uploading segments... Aug 7 23:57:22.981169317 (14h17m57.496271889s) [upload=14690]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:23.078169780 (14h17m57.593272348s) [upload=14705]: Starting upload Aug 7 23:57:23.078287503 (14h17m57.593390079s) [upload=14705]: Done starting upload Aug 7 23:57:23.078353530 (14h17m57.593456107s) [upload=14705]: Uploading segments... Aug 7 23:57:23.078394403 (14h17m57.593496992s) [upload=14705]: Got next segment. Inline: true Aug 7 23:57:23.078400888 (14h17m57.593503457s) [upload=14705]: Waiting for error group managing segments... Aug 7 23:57:23.078452630 (14h17m57.593555201s) [upload=14705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:23.078457270 (14h17m57.593559838s) [upload=14705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:23.078460483 (14h17m57.593563048s) [upload=14705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:23.049396934 (14h18m57.564499518s) [upload=14705]: Done uploading segments... Aug 7 23:58:23.049418059 (14h18m57.564520634s) [upload=14705]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:23.169720285 (14h18m57.684822852s) [upload=14722]: Starting upload Aug 7 23:58:23.169779212 (14h18m57.684881788s) [upload=14722]: Done starting upload Aug 7 23:58:23.169840834 (14h18m57.684943401s) [upload=14722]: Uploading segments... Aug 7 23:58:23.169880437 (14h18m57.684983021s) [upload=14722]: Got next segment. Inline: true Aug 7 23:58:23.169885772 (14h18m57.684988338s) [upload=14722]: Waiting for error group managing segments... Aug 7 23:58:23.169927728 (14h18m57.685030303s) [upload=14722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:23.169933919 (14h18m57.685036498s) [upload=14722]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:23.169938117 (14h18m57.685040704s) [upload=14722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:23.138140642 (14h19m57.653243209s) [upload=14722]: Done uploading segments... Aug 7 23:59:23.138159469 (14h19m57.653262035s) [upload=14722]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:35.502179618 (14h20m10.017282184s) [upload=14745]: Starting upload Aug 7 23:59:35.502246881 (14h20m10.017349449s) [upload=14745]: Done starting upload Aug 7 23:59:35.502308502 (14h20m10.017411071s) [upload=14745]: Uploading segments... Aug 7 23:59:35.502347394 (14h20m10.017449979s) [upload=14745]: Got next segment. Inline: true Aug 7 23:59:35.502351750 (14h20m10.017454327s) [upload=14745]: Waiting for error group managing segments... Aug 7 23:59:35.502390443 (14h20m10.017493009s) [upload=14745]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:35.502394821 (14h20m10.017497398s) [upload=14745]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:35.502398119 (14h20m10.017500695s) [upload=14745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:35.472447782 (14h21m9.987550364s) [upload=14745]: Done uploading segments... Aug 8 00:00:35.472473966 (14h21m9.987576558s) [upload=14745]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:00:35.593243192 (14h21m10.10834576s) [upload=14760]: Starting upload Aug 8 00:00:35.593395835 (14h21m10.108498415s) [upload=14760]: Done starting upload Aug 8 00:00:35.593497926 (14h21m10.10860049s) [upload=14760]: Uploading segments... Aug 8 00:00:35.593659481 (14h21m10.108762045s) [upload=14760]: Got next segment. Inline: true Aug 8 00:00:35.593667686 (14h21m10.108770273s) [upload=14760]: Waiting for error group managing segments... Aug 8 00:00:35.593750345 (14h21m10.10885293s) [upload=14760]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:35.593773930 (14h21m10.108876508s) [upload=14760]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:35.593782409 (14h21m10.108884991s) [upload=14760]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:35.662793141 (14h21m10.177895709s) [upload=14760]: Done uploading segments... Aug 8 00:00:35.662815087 (14h21m10.177917658s) [upload=14760]: Upload finished. err: <nil> Aug 8 00:01:23.648690375 (14h21m58.163792959s) [upload=14775]: Starting upload Aug 8 00:01:23.648799550 (14h21m58.163902126s) [upload=14775]: Done starting upload Aug 8 00:01:23.648928369 (14h21m58.164030934s) [upload=14775]: Uploading segments... Aug 8 00:01:23.648986999 (14h21m58.164089578s) [upload=14775]: Got next segment. Inline: true Aug 8 00:01:23.648992059 (14h21m58.164094634s) [upload=14775]: Waiting for error group managing segments... Aug 8 00:01:23.649037148 (14h21m58.164139714s) [upload=14775]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:23.649041722 (14h21m58.164144298s) [upload=14775]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:23.649044950 (14h21m58.164147527s) [upload=14775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:23.618855090 (14h22m58.133957671s) [upload=14775]: Done uploading segments... Aug 8 00:02:23.618900564 (14h22m58.134003142s) [upload=14775]: Upload finished. err: metaclient: context deadline exceeded Duration: 243.309788ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 10 out of 60 is failed (threshold=3) [XXX...X.X......................................X.XXX...X....] duration: 1.141815483s FAIL (2025-08-08 00:06:09 UTC) operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 474260 (14h23m36.87657683s) [upload=14802]: Got next segment. Inline: false Aug 8 00:03:02.361540620 (14h23m36.876643199s) [upload=14802]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:03:02.361546434 (14h23m36.87664901s) [upload=14802]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:03:57.203920880 (14h24m31.719023446s) [seg_pos={0 0}, upload=14802]: Begin upload segment... Aug 8 00:03:57.203988004 (14h24m31.719090583s) [seg_pos={0 0}, upload=14802]: Done begin upload segment. Aug 8 00:03:57.204020830 (14h24m31.719123417s) [noise=true , address=45.91.116.10:4107 , piece_id=XM3BQK4U7EPQ4VIWN6I44B5U25TDUXWEK2TEFNT7FDNCBKU6DTWA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204041644 (14h24m31.71914423s) [noise=true , address=104.36.82.23:4449 , piece_id=TAFC4ZPBW4MY6RC5TX4CAUPBCJMHXORTGIXDID23ICWC43R6NPWQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204477526 (14h24m31.719580107s) [noise=true , address=79.127.201.215:4346 , piece_id=GMA63JIOTMY7O66ASAO35CP25HFTLKYXVT3HHIJE3GGJJ2WZT4RA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204519100 (14h24m31.719621687s) [noise=true , address=104.36.82.9:4022 , piece_id=N7RQZECLABTXW3N6O4K6CY245IGHUUPGSHLNIENEGDXA2KZUNHGQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204895465 (14h24m31.719998052s) [noise=true , address=104.36.82.38:4700 , piece_id=BERTXTXRJUHPIWKAIPEG4GYJ355BDN6K7UFF3AEYW2YMGQWFEJKQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204925306 (14h24m31.720027871s) [noise=true , address=64.49.13.27:4960 , piece_id=LAJD7GW4V4WEKS5EMUZ2XSL4T2VEL74LBL2PXQF2HE5TFZM2W5JQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.205328523 (14h24m31.720431105s) [noise=true , address=104.36.82.47:4545 , piece_id=QZTD52366FHJIE36E2BLHPZO2Q2EUONODVNHY3NKXLYMNPFTP5TQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.205769139 (14h24m31.720871721s) [noise=true , address=104.36.82.5:4854 , piece_id=C55AYUS7YLDCAWRE6EUWGEAJFBPME2N5IFK4PUJJT6XCFFVCD7OQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.207436252 (14h24m31.722538839s) [noise=true , address=45.91.116.6:4008 , piece_id=OMLZN2SLPT4SNYRDIUTCVMOERA4ALP6YP4QS4AVIG66VRWQNNGNQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.207828410 (14h24m31.722930995s) [noise=true , address=104.36.82.36:4629 , piece_id=3SZDIUDWSAZAWWG4BHVGGAQPU7Q7XWWIQWOKTROV6NPOWUALK5BA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.208468343 (14h24m31.723570931s) [noise=true , address=77.90.31.14:4684 , piece_id=EVAAF4LC6E3JVN6AOBWPCXTP4ILJSQDJP47ZFJHAYPGEO5KEA5MQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.209378344 (14h24m31.72448093s) [noise=true , address=64.49.13.154:4927 , piece_id=GUE6TO7SOONGAQ3UWXEAMDVBN2PDRKYTNADRTH6FWPIPKVEX5JOQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.210191405 (14h24m31.72529399s) [noise=true , address=104.36.82.43:4853 , piece_id=M5GL6IYJ2XTIJRCPWYBISDDYC33OUNTY6RRAKWP374PJQT33AVTQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.211289253 (14h24m31.726391837s) [noise=true , address=104.36.82.46:4509 , piece_id=IYIMDZG6RGFTWDNZYHSZ6FNGN6CGVXNF6VQNXSDUG3US7VSNN7UA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.211675380 (14h24m31.726777962s) [noise=true , address=104.36.82.3:4794 , piece_id=6URKBCOCEX2K3ZROKDT5GI24ZP642SB7WYCU4IYOPQ2XWMHHXMTQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.212135467 (14h24m31.727238053s) [noise=true , address=193.149.164.143:4695 , piece_id=3QVTVIGKIEY4QZWZOUDAQWSKROHOKNR6XTG4DXXUWIVJMNWQEIZA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.212506215 (14h24m31.727608796s) [noise=true , address=104.36.82.34:4565 , piece_id=Q5YZ7G72VW7QBR64RLS2HMQX5CU3L5AELKGZ5Q3J74EPOGSBGNWA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.212830381 (14h24m31.727932969s) [noise=true , address=104.36.82.62:4421 , piece_id=XSTGRY73EHC33KBNGDTYUZOSLBQS5Z3ZY6QOPAEDER4NTRPG3B3Q, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.294294492 (14h24m31.809397074s) [upload=14802]: Next returned nil segment Aug 8 00:03:57.294311485 (14h24m31.809414071s) [upload=14802]: Waiting for error group managing segments... Aug 8 00:03:57.306352633 (14h24m31.821455213s) [noise=true , address=104.36.82.34:4565 , piece_id=Q5YZ7G72VW7QBR64RLS2HMQX5CU3L5AELKGZ5Q3J74EPOGSBGNWA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.307346446 (14h24m31.822449009s) [noise=true , address=45.91.116.6:4008 , piece_id=OMLZN2SLPT4SNYRDIUTCVMOERA4ALP6YP4QS4AVIG66VRWQNNGNQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.308700400 (14h24m31.823802978s) [noise=true , address=104.36.82.36:4629 , piece_id=3SZDIUDWSAZAWWG4BHVGGAQPU7Q7XWWIQWOKTROV6NPOWUALK5BA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.312160131 (14h24m31.827262707s) [noise=true , address=104.36.82.62:4421 , piece_id=XSTGRY73EHC33KBNGDTYUZOSLBQS5Z3ZY6QOPAEDER4NTRPG3B3Q, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.320102036 (14h24m31.835204602s) [noise=true , address=104.36.82.47:4545 , piece_id=QZTD52366FHJIE36E2BLHPZO2Q2EUONODVNHY3NKXLYMNPFTP5TQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.321164433 (14h24m31.836267014s) [noise=true , address=45.91.116.10:4107 , piece_id=XM3BQK4U7EPQ4VIWN6I44B5U25TDUXWEK2TEFNT7FDNCBKU6DTWA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.326337358 (14h24m31.841439935s) [noise=true , address=77.90.31.14:4684 , piece_id=EVAAF4LC6E3JVN6AOBWPCXTP4ILJSQDJP47ZFJHAYPGEO5KEA5MQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.352532224 (14h24m31.867634786s) [noise=true , address=104.36.82.23:4449 , piece_id=TAFC4ZPBW4MY6RC5TX4CAUPBCJMHXORTGIXDID23ICWC43R6NPWQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.356261323 (14h24m31.871363886s) [noise=true , address=104.36.82.43:4853 , piece_id=M5GL6IYJ2XTIJRCPWYBISDDYC33OUNTY6RRAKWP374PJQT33AVTQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.423088525 (14h24m31.938191096s) [noise=true , address=104.36.82.38:4700 , piece_id=BERTXTXRJUHPIWKAIPEG4GYJ355BDN6K7UFF3AEYW2YMGQWFEJKQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.471455369 (14h24m31.986557939s) [noise=true , address=79.127.201.215:4346 , piece_id=GMA63JIOTMY7O66ASAO35CP25HFTLKYXVT3HHIJE3GGJJ2WZT4RA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.749384271 (14h24m32.264486838s) [noise=true , address=104.36.82.9:4022 , piece_id=N7RQZECLABTXW3N6O4K6CY245IGHUUPGSHLNIENEGDXA2KZUNHGQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.873513358 (14h24m32.388615936s) [noise=true , address=193.149.164.143:4695 , piece_id=3QVTVIGKIEY4QZWZOUDAQWSKROHOKNR6XTG4DXXUWIVJMNWQEIZA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.876598294 (14h24m32.391700872s) [noise=true , address=64.49.13.27:4960 , piece_id=LAJD7GW4V4WEKS5EMUZ2XSL4T2VEL74LBL2PXQF2HE5TFZM2W5JQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.876652272 (14h24m32.391754851s) [seg_pos={0 0}, upload=14802]: Segment reached optimal threshold of 14 pieces. Aug 8 00:03:57.877023576 (14h24m32.392126154s) [noise=true , address=104.36.82.5:4854 , piece_id=C55AYUS7YLDCAWRE6EUWGEAJFBPME2N5IFK4PUJJT6XCFFVCD7OQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19pxQXokdTJ9W6k5aZrzGZSoAn9p1NqpwbTWMSC6U7ncr8ipwy): protocol: write tcp 159.223.143.39:37798->104.36.82.5:4854: use of closed network connection; write tcp 159.223.143.39:37798->104.36.82.5:4854: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37798->104.36.82.5:4854: use of closed network connection Aug 8 00:03:57.877160473 (14h24m32.392263043s) [noise=true , address=64.49.13.154:4927 , piece_id=GUE6TO7SOONGAQ3UWXEAMDVBN2PDRKYTNADRTH6FWPIPKVEX5JOQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12U8eKLnBHxvuutyABVGSgUJbG3kr9Cce4sDx59LQiJHT1W2U8k): protocol: expected piece hash; context canceled; EOF Aug 8 00:03:57.877284905 (14h24m32.392387475s) [noise=true , address=104.36.82.46:4509 , piece_id=IYIMDZG6RGFTWDNZYHSZ6FNGN6CGVXNF6VQNXSDUG3US7VSNN7UA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y2NmauNvCyrMJ5z66Lihb3h3C9nwdJYqfSfNmC5dwDGYQqUKh): protocol: expected piece hash; context canceled; EOF Aug 8 00:03:57.877449504 (14h24m32.392552073s) [noise=true , address=104.36.82.3:4794 , piece_id=6URKBCOCEX2K3ZROKDT5GI24ZP642SB7WYCU4IYOPQ2XWMHHXMTQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ctLLD5skk4t5i3BFYgoTmuZ2aqsX23wDGDMnKriQ7U2KXyPAo): protocol: write tcp 159.223.143.39:47368->104.36.82.3:4794: use of closed network connection; context canceled Aug 8 00:03:57.877469652 (14h24m32.392572221s) [seg_pos={0 0}, upload=14802]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:03:57.877571490 (14h24m32.392674067s) [upload=14802]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:03:57.877576879 (14h24m32.392679458s) [upload=14802]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:04:02.023423762 (14h24m36.538526336s) [upload=14802]: Done uploading segments... Aug 8 00:04:02.023447427 (14h24m36.538550004s) [upload=14802]: Upload finished. err: <nil> Duration: 1.141841564s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 32.57356ms target: https://us1.storj.io/login FAIL (2025-08-08 00:06:07 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 32.601633ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [........................................XXX.X...XX.XX.......] duration: 600.851666ms FAIL (2025-08-08 00:05:57 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 0.31.10:4517 , piece_id=4XNCSILTEXFXI3T5YJLDOHWW2766ZY2YM723VBVF7XYPPTZWBKEA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.130393931 (14h19m30.645496508s) [noise=true , address=45.91.116.8:4048 , piece_id=DGULAYZHOXDN4IO3EHSUHYN3Z3GOMWRPSF3IF762Z7PQNDBWWUCA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.130898433 (14h19m30.646001001s) [noise=true , address=104.36.82.11:4115 , piece_id=BKAK43Z32DE3HOXYPSW4BH5LUWMQ3TINDTPESBBXOBHA6NDDCQSA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.131654285 (14h19m30.646756854s) [noise=true , address=77.90.31.16:4733 , piece_id=R6AKPWEKFLEI7NP4XAMYL72SHIKNIORWP4JXFZXLVRTVIOXWGF2Q, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.132299167 (14h19m30.647401729s) [noise=true , address=45.91.116.13:4153 , piece_id=G35ZWVBI2OZYXYXOY5RKDG6IO2G4PET4BDFHGLQV4NVTUVAGVNJQ, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.132815896 (14h19m30.647918459s) [noise=true , address=77.90.31.17:4772 , piece_id=76IYY7NQGZ4WZTPRZ5MW7B7ABMNNQJGHKGZHPKLCX4KHAWNJ7XGA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.133305230 (14h19m30.648407796s) [noise=true , address=104.36.82.6:4875 , piece_id=RHXGB245P3CNRWQJPKBZ4RL6BV5AGIHV35TVLDCWIUVCVU2TRZJA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.134064921 (14h19m30.649167503s) [noise=true , address=104.36.82.2:4731 , piece_id=374ZMGN47ZFQDXM2NI52ZLYMNYNUMB5HD7UZWKFGEKMDDAG4DUAA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.134501761 (14h19m30.649604335s) [noise=true , address=104.36.82.5:4840 , piece_id=JKJE7FIIKVDBL2MQF7IWVNMYH63QV72QXJOYARECFS5FJMSAOK4A, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.135039838 (14h19m30.650142405s) [noise=true , address=45.91.116.14:4191 , piece_id=5SWPQ2TBXYVYNFOGMH6QVANJVGNFEVOJ6UFMAX74DCC2WWKRSBNA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.135450095 (14h19m30.650552665s) [noise=true , address=77.90.31.19:4842 , piece_id=OUTUWUFXDDTLL52UJ6RTLORFEPWMQVL6JYBP3PPFY2DHS7JQHMBQ, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.135750292 (14h19m30.650852861s) [noise=true , address=77.90.31.15:4697 , piece_id=7MTL64JADCQRGS6UHDTT2ZGJFJV26TP4VYEPVM43MFLNXPPHFRQA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.136062113 (14h19m30.651164675s) [noise=true , address=104.36.82.9:4005 , piece_id=BJR6LEV57IPYIKASKNYGCMIKM7XMPZ6J3ASRB4B2ZZHAD3WQB47Q, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.136413662 (14h19m30.65151623s) [noise=true , address=45.91.116.9:4263 , piece_id=3G4QVYQG6AJ555B632PNRZWUUJM23K3GWSIPGNWJBCQR3ZZGZ4BQ, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.136771535 (14h19m30.651874107s) [noise=true , address=104.36.82.16:4295 , piece_id=QFJ444VYPVNGDMZ2WO2MBWMUH2U6M5K6TQ66T6SHQJW2CM245BTA, seg_pos={0 0}, upload=14734]: Uploading piece... Aug 7 23:58:56.209688234 (14h19m30.724790803s) [upload=14734]: Next returned nil segment Aug 7 23:58:56.209703014 (14h19m30.72480559s) [upload=14734]: Waiting for error group managing segments... Aug 7 23:58:56.214389537 (14h19m30.729492107s) [noise=true , address=45.91.116.9:4263 , piece_id=3G4QVYQG6AJ555B632PNRZWUUJM23K3GWSIPGNWJBCQR3ZZGZ4BQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.214445477 (14h19m30.729548054s) [noise=true , address=45.91.116.13:4153 , piece_id=G35ZWVBI2OZYXYXOY5RKDG6IO2G4PET4BDFHGLQV4NVTUVAGVNJQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.214979544 (14h19m30.73008212s) [noise=true , address=77.90.31.19:4842 , piece_id=OUTUWUFXDDTLL52UJ6RTLORFEPWMQVL6JYBP3PPFY2DHS7JQHMBQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.215082462 (14h19m30.730185028s) [noise=true , address=45.91.116.14:4191 , piece_id=5SWPQ2TBXYVYNFOGMH6QVANJVGNFEVOJ6UFMAX74DCC2WWKRSBNA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.215826575 (14h19m30.730929141s) [noise=true , address=77.90.31.10:4517 , piece_id=4XNCSILTEXFXI3T5YJLDOHWW2766ZY2YM723VBVF7XYPPTZWBKEA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.215911546 (14h19m30.731014112s) [noise=true , address=77.90.31.16:4733 , piece_id=R6AKPWEKFLEI7NP4XAMYL72SHIKNIORWP4JXFZXLVRTVIOXWGF2Q, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.216710632 (14h19m30.731813201s) [noise=true , address=77.90.31.18:4807 , piece_id=B3PSWCRJNGDMD6LDDM436N3X75NAA4DEEPLDKL2GTBBJBG3I6ZSQ, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.217564442 (14h19m30.732667019s) [noise=true , address=77.90.31.15:4697 , piece_id=7MTL64JADCQRGS6UHDTT2ZGJFJV26TP4VYEPVM43MFLNXPPHFRQA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.218100336 (14h19m30.733202913s) [noise=true , address=77.90.31.17:4772 , piece_id=76IYY7NQGZ4WZTPRZ5MW7B7ABMNNQJGHKGZHPKLCX4KHAWNJ7XGA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.220843127 (14h19m30.735945696s) [noise=true , address=104.36.82.11:4115 , piece_id=BKAK43Z32DE3HOXYPSW4BH5LUWMQ3TINDTPESBBXOBHA6NDDCQSA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.221894363 (14h19m30.736996944s) [noise=true , address=104.36.82.6:4875 , piece_id=RHXGB245P3CNRWQJPKBZ4RL6BV5AGIHV35TVLDCWIUVCVU2TRZJA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.227829104 (14h19m30.742931668s) [noise=true , address=45.91.116.8:4048 , piece_id=DGULAYZHOXDN4IO3EHSUHYN3Z3GOMWRPSF3IF762Z7PQNDBWWUCA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.326406182 (14h19m30.841508752s) [noise=true , address=104.36.82.9:4005 , piece_id=BJR6LEV57IPYIKASKNYGCMIKM7XMPZ6J3ASRB4B2ZZHAD3WQB47Q, seg_pos={0 0}, upload=14734]: Done uploading piece. err: <nil> Aug 7 23:58:56.326453057 (14h19m30.841555634s) [seg_pos={0 0}, upload=14734]: Segment reached optimal threshold of 13 pieces. Aug 7 23:58:56.326683605 (14h19m30.841786189s) [noise=true , address=104.36.82.2:4731 , piece_id=374ZMGN47ZFQDXM2NI52ZLYMNYNUMB5HD7UZWKFGEKMDDAG4DUAA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Aug 7 23:58:56.326786377 (14h19m30.84188896s) [noise=true , address=104.36.82.5:4840 , piece_id=JKJE7FIIKVDBL2MQF7IWVNMYH63QV72QXJOYARECFS5FJMSAOK4A, seg_pos={0 0}, upload=14734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: write tcp 159.223.143.39:60496->104.36.82.5:4840: use of closed network connection; write tcp 159.223.143.39:60496->104.36.82.5:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60496->104.36.82.5:4840: use of closed network connection Aug 7 23:58:56.326888961 (14h19m30.841991543s) [noise=true , address=104.36.82.16:4295 , piece_id=QFJ444VYPVNGDMZ2WO2MBWMUH2U6M5K6TQ66T6SHQJW2CM245BTA, seg_pos={0 0}, upload=14734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: read tcp 159.223.143.39:34618->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Aug 7 23:58:56.326913735 (14h19m30.842016315s) [seg_pos={0 0}, upload=14734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 23:58:56.326998778 (14h19m30.842101358s) [upload=14734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 23:58:56.327004732 (14h19m30.84210731s) [upload=14734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:56.375303093 (14h19m30.89040566s) [upload=14734]: Done uploading segments... Aug 7 23:58:56.375321956 (14h19m30.890424523s) [upload=14734]: Upload finished. err: <nil> Aug 7 23:59:26.492160234 (14h20m1.007262811s) [upload=14742]: Starting upload Aug 7 23:59:26.492250238 (14h20m1.007352804s) [upload=14742]: Done starting upload Aug 7 23:59:26.492323252 (14h20m1.00742583s) [upload=14742]: Uploading segments... Aug 7 23:59:26.492605785 (14h20m1.007708349s) [upload=14742]: Got next segment. Inline: false Aug 7 23:59:26.492681402 (14h20m1.007783987s) [upload=14742]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:26.492689520 (14h20m1.007792101s) [upload=14742]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 23:59:56.459090933 (14h20m30.974193512s) [upload=14742]: Next segment err: context deadline exceeded Aug 7 23:59:56.459165231 (14h20m30.974267796s) [upload=14742]: Done uploading segments... Aug 7 23:59:56.459181138 (14h20m30.974283703s) [upload=14742]: Upload finished. err: context deadline exceeded Aug 7 23:59:56.572495425 (14h20m31.087597994s) [upload=14751]: Starting upload Aug 7 23:59:56.572578962 (14h20m31.087681544s) [upload=14751]: Done starting upload Aug 7 23:59:56.572818459 (14h20m31.087921029s) [upload=14751]: Uploading segments... Aug 7 23:59:56.572953386 (14h20m31.088055953s) [upload=14751]: Got next segment. Inline: false Aug 7 23:59:56.573048904 (14h20m31.088151483s) [upload=14751]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:56.573066041 (14h20m31.088168618s) [upload=14751]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:00:26.529103664 (14h21m1.044206252s) [upload=14751]: Next segment err: context deadline exceeded Aug 8 00:00:26.529241226 (14h21m1.044343811s) [upload=14751]: Done uploading segments... Aug 8 00:00:26.529265086 (14h21m1.044367654s) [upload=14751]: Upload finished. err: context deadline exceeded Duration: 600.880723ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.X.....................................X..........X........] duration: 898.646292ms FAIL (2025-08-08 00:05:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 898.669433ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [........................................X..X....XX...X......] duration: 317.818763ms FAIL (2025-08-08 00:05:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 317.88351ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [........................................X..XXX..X...X.......] duration: 219.445329ms FAIL (2025-08-08 00:05:46 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 219.473172ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 11 out of 60 is failed (threshold=3) [XXXX...X.X......................................X.XXX...X...] duration: 1.38136646s FAIL (2025-08-08 00:05:45 UTC) operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 474260 (14h23m36.87657683s) [upload=14802]: Got next segment. Inline: false Aug 8 00:03:02.361540620 (14h23m36.876643199s) [upload=14802]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:03:02.361546434 (14h23m36.87664901s) [upload=14802]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 8 00:03:57.203920880 (14h24m31.719023446s) [seg_pos={0 0}, upload=14802]: Begin upload segment... Aug 8 00:03:57.203988004 (14h24m31.719090583s) [seg_pos={0 0}, upload=14802]: Done begin upload segment. Aug 8 00:03:57.204020830 (14h24m31.719123417s) [noise=true , address=45.91.116.10:4107 , piece_id=XM3BQK4U7EPQ4VIWN6I44B5U25TDUXWEK2TEFNT7FDNCBKU6DTWA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204041644 (14h24m31.71914423s) [noise=true , address=104.36.82.23:4449 , piece_id=TAFC4ZPBW4MY6RC5TX4CAUPBCJMHXORTGIXDID23ICWC43R6NPWQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204477526 (14h24m31.719580107s) [noise=true , address=79.127.201.215:4346 , piece_id=GMA63JIOTMY7O66ASAO35CP25HFTLKYXVT3HHIJE3GGJJ2WZT4RA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204519100 (14h24m31.719621687s) [noise=true , address=104.36.82.9:4022 , piece_id=N7RQZECLABTXW3N6O4K6CY245IGHUUPGSHLNIENEGDXA2KZUNHGQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204895465 (14h24m31.719998052s) [noise=true , address=104.36.82.38:4700 , piece_id=BERTXTXRJUHPIWKAIPEG4GYJ355BDN6K7UFF3AEYW2YMGQWFEJKQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.204925306 (14h24m31.720027871s) [noise=true , address=64.49.13.27:4960 , piece_id=LAJD7GW4V4WEKS5EMUZ2XSL4T2VEL74LBL2PXQF2HE5TFZM2W5JQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.205328523 (14h24m31.720431105s) [noise=true , address=104.36.82.47:4545 , piece_id=QZTD52366FHJIE36E2BLHPZO2Q2EUONODVNHY3NKXLYMNPFTP5TQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.205769139 (14h24m31.720871721s) [noise=true , address=104.36.82.5:4854 , piece_id=C55AYUS7YLDCAWRE6EUWGEAJFBPME2N5IFK4PUJJT6XCFFVCD7OQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.207436252 (14h24m31.722538839s) [noise=true , address=45.91.116.6:4008 , piece_id=OMLZN2SLPT4SNYRDIUTCVMOERA4ALP6YP4QS4AVIG66VRWQNNGNQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.207828410 (14h24m31.722930995s) [noise=true , address=104.36.82.36:4629 , piece_id=3SZDIUDWSAZAWWG4BHVGGAQPU7Q7XWWIQWOKTROV6NPOWUALK5BA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.208468343 (14h24m31.723570931s) [noise=true , address=77.90.31.14:4684 , piece_id=EVAAF4LC6E3JVN6AOBWPCXTP4ILJSQDJP47ZFJHAYPGEO5KEA5MQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.209378344 (14h24m31.72448093s) [noise=true , address=64.49.13.154:4927 , piece_id=GUE6TO7SOONGAQ3UWXEAMDVBN2PDRKYTNADRTH6FWPIPKVEX5JOQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.210191405 (14h24m31.72529399s) [noise=true , address=104.36.82.43:4853 , piece_id=M5GL6IYJ2XTIJRCPWYBISDDYC33OUNTY6RRAKWP374PJQT33AVTQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.211289253 (14h24m31.726391837s) [noise=true , address=104.36.82.46:4509 , piece_id=IYIMDZG6RGFTWDNZYHSZ6FNGN6CGVXNF6VQNXSDUG3US7VSNN7UA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.211675380 (14h24m31.726777962s) [noise=true , address=104.36.82.3:4794 , piece_id=6URKBCOCEX2K3ZROKDT5GI24ZP642SB7WYCU4IYOPQ2XWMHHXMTQ, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.212135467 (14h24m31.727238053s) [noise=true , address=193.149.164.143:4695 , piece_id=3QVTVIGKIEY4QZWZOUDAQWSKROHOKNR6XTG4DXXUWIVJMNWQEIZA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.212506215 (14h24m31.727608796s) [noise=true , address=104.36.82.34:4565 , piece_id=Q5YZ7G72VW7QBR64RLS2HMQX5CU3L5AELKGZ5Q3J74EPOGSBGNWA, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.212830381 (14h24m31.727932969s) [noise=true , address=104.36.82.62:4421 , piece_id=XSTGRY73EHC33KBNGDTYUZOSLBQS5Z3ZY6QOPAEDER4NTRPG3B3Q, seg_pos={0 0}, upload=14802]: Uploading piece... Aug 8 00:03:57.294294492 (14h24m31.809397074s) [upload=14802]: Next returned nil segment Aug 8 00:03:57.294311485 (14h24m31.809414071s) [upload=14802]: Waiting for error group managing segments... Aug 8 00:03:57.306352633 (14h24m31.821455213s) [noise=true , address=104.36.82.34:4565 , piece_id=Q5YZ7G72VW7QBR64RLS2HMQX5CU3L5AELKGZ5Q3J74EPOGSBGNWA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.307346446 (14h24m31.822449009s) [noise=true , address=45.91.116.6:4008 , piece_id=OMLZN2SLPT4SNYRDIUTCVMOERA4ALP6YP4QS4AVIG66VRWQNNGNQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.308700400 (14h24m31.823802978s) [noise=true , address=104.36.82.36:4629 , piece_id=3SZDIUDWSAZAWWG4BHVGGAQPU7Q7XWWIQWOKTROV6NPOWUALK5BA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.312160131 (14h24m31.827262707s) [noise=true , address=104.36.82.62:4421 , piece_id=XSTGRY73EHC33KBNGDTYUZOSLBQS5Z3ZY6QOPAEDER4NTRPG3B3Q, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.320102036 (14h24m31.835204602s) [noise=true , address=104.36.82.47:4545 , piece_id=QZTD52366FHJIE36E2BLHPZO2Q2EUONODVNHY3NKXLYMNPFTP5TQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.321164433 (14h24m31.836267014s) [noise=true , address=45.91.116.10:4107 , piece_id=XM3BQK4U7EPQ4VIWN6I44B5U25TDUXWEK2TEFNT7FDNCBKU6DTWA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.326337358 (14h24m31.841439935s) [noise=true , address=77.90.31.14:4684 , piece_id=EVAAF4LC6E3JVN6AOBWPCXTP4ILJSQDJP47ZFJHAYPGEO5KEA5MQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.352532224 (14h24m31.867634786s) [noise=true , address=104.36.82.23:4449 , piece_id=TAFC4ZPBW4MY6RC5TX4CAUPBCJMHXORTGIXDID23ICWC43R6NPWQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.356261323 (14h24m31.871363886s) [noise=true , address=104.36.82.43:4853 , piece_id=M5GL6IYJ2XTIJRCPWYBISDDYC33OUNTY6RRAKWP374PJQT33AVTQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.423088525 (14h24m31.938191096s) [noise=true , address=104.36.82.38:4700 , piece_id=BERTXTXRJUHPIWKAIPEG4GYJ355BDN6K7UFF3AEYW2YMGQWFEJKQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.471455369 (14h24m31.986557939s) [noise=true , address=79.127.201.215:4346 , piece_id=GMA63JIOTMY7O66ASAO35CP25HFTLKYXVT3HHIJE3GGJJ2WZT4RA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.749384271 (14h24m32.264486838s) [noise=true , address=104.36.82.9:4022 , piece_id=N7RQZECLABTXW3N6O4K6CY245IGHUUPGSHLNIENEGDXA2KZUNHGQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.873513358 (14h24m32.388615936s) [noise=true , address=193.149.164.143:4695 , piece_id=3QVTVIGKIEY4QZWZOUDAQWSKROHOKNR6XTG4DXXUWIVJMNWQEIZA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.876598294 (14h24m32.391700872s) [noise=true , address=64.49.13.27:4960 , piece_id=LAJD7GW4V4WEKS5EMUZ2XSL4T2VEL74LBL2PXQF2HE5TFZM2W5JQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: <nil> Aug 8 00:03:57.876652272 (14h24m32.391754851s) [seg_pos={0 0}, upload=14802]: Segment reached optimal threshold of 14 pieces. Aug 8 00:03:57.877023576 (14h24m32.392126154s) [noise=true , address=104.36.82.5:4854 , piece_id=C55AYUS7YLDCAWRE6EUWGEAJFBPME2N5IFK4PUJJT6XCFFVCD7OQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19pxQXokdTJ9W6k5aZrzGZSoAn9p1NqpwbTWMSC6U7ncr8ipwy): protocol: write tcp 159.223.143.39:37798->104.36.82.5:4854: use of closed network connection; write tcp 159.223.143.39:37798->104.36.82.5:4854: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37798->104.36.82.5:4854: use of closed network connection Aug 8 00:03:57.877160473 (14h24m32.392263043s) [noise=true , address=64.49.13.154:4927 , piece_id=GUE6TO7SOONGAQ3UWXEAMDVBN2PDRKYTNADRTH6FWPIPKVEX5JOQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12U8eKLnBHxvuutyABVGSgUJbG3kr9Cce4sDx59LQiJHT1W2U8k): protocol: expected piece hash; context canceled; EOF Aug 8 00:03:57.877284905 (14h24m32.392387475s) [noise=true , address=104.36.82.46:4509 , piece_id=IYIMDZG6RGFTWDNZYHSZ6FNGN6CGVXNF6VQNXSDUG3US7VSNN7UA, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y2NmauNvCyrMJ5z66Lihb3h3C9nwdJYqfSfNmC5dwDGYQqUKh): protocol: expected piece hash; context canceled; EOF Aug 8 00:03:57.877449504 (14h24m32.392552073s) [noise=true , address=104.36.82.3:4794 , piece_id=6URKBCOCEX2K3ZROKDT5GI24ZP642SB7WYCU4IYOPQ2XWMHHXMTQ, seg_pos={0 0}, upload=14802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ctLLD5skk4t5i3BFYgoTmuZ2aqsX23wDGDMnKriQ7U2KXyPAo): protocol: write tcp 159.223.143.39:47368->104.36.82.3:4794: use of closed network connection; context canceled Aug 8 00:03:57.877469652 (14h24m32.392572221s) [seg_pos={0 0}, upload=14802]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 8 00:03:57.877571490 (14h24m32.392674067s) [upload=14802]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 8 00:03:57.877576879 (14h24m32.392679458s) [upload=14802]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:04:02.023423762 (14h24m36.538526336s) [upload=14802]: Done uploading segments... Aug 8 00:04:02.023447427 (14h24m36.538550004s) [upload=14802]: Upload finished. err: <nil> Duration: 1.38141799s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 34.113452ms target: https://us1.storj.io/login FAIL (2025-08-08 00:05:45 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.150.199.48:443: connect: connection refused Duration: 34.137593ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 8 out of 60 is failed (threshold=3) [.............................................XXXXXXX.X......] duration: 215.69625ms FAIL (2025-08-08 00:05:42 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ne starting upload Aug 7 23:52:04.519569621 (14h12m39.034672186s) [upload=14619]: Uploading segments... Aug 7 23:52:04.519675504 (14h12m39.034778091s) [upload=14619]: Got next segment. Inline: true Aug 7 23:52:04.519701072 (14h12m39.034803649s) [upload=14619]: Waiting for error group managing segments... Aug 7 23:52:04.519755555 (14h12m39.034858131s) [upload=14619]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:52:04.519760650 (14h12m39.034863226s) [upload=14619]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:52:04.519777078 (14h12m39.034879654s) [upload=14619]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:52:04.588422656 (14h12m39.103525241s) [upload=14619]: Done uploading segments... Aug 7 23:52:04.588444719 (14h12m39.103547297s) [upload=14619]: Upload finished. err: <nil> Aug 7 23:53:08.147087231 (14h13m42.662189804s) [upload=14641]: Starting upload Aug 7 23:53:08.147181411 (14h13m42.662283996s) [upload=14641]: Done starting upload Aug 7 23:53:08.147281202 (14h13m42.66238377s) [upload=14641]: Uploading segments... Aug 7 23:53:08.147322830 (14h13m42.662425415s) [upload=14641]: Got next segment. Inline: true Aug 7 23:53:08.147327701 (14h13m42.662430277s) [upload=14641]: Waiting for error group managing segments... Aug 7 23:53:08.147455761 (14h13m42.66255833s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:53:08.147460709 (14h13m42.662563285s) [upload=14641]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:53:08.147463890 (14h13m42.662566457s) [upload=14641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:54:08.116969456 (14h14m42.632072039s) [upload=14641]: Done uploading segments... Aug 7 23:54:08.117007036 (14h14m42.632109618s) [upload=14641]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:54:22.831162336 (14h14m57.346264913s) [upload=14661]: Starting upload Aug 7 23:54:22.831234431 (14h14m57.346337008s) [upload=14661]: Done starting upload Aug 7 23:54:22.831306794 (14h14m57.346409362s) [upload=14661]: Uploading segments... Aug 7 23:54:22.831343257 (14h14m57.346445842s) [upload=14661]: Got next segment. Inline: true Aug 7 23:54:22.831347808 (14h14m57.346450375s) [upload=14661]: Waiting for error group managing segments... Aug 7 23:54:22.831399492 (14h14m57.346502071s) [upload=14661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:54:22.831404349 (14h14m57.346506915s) [upload=14661]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:54:22.831407696 (14h14m57.346510265s) [upload=14661]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:55:22.799123829 (14h15m57.314226409s) [upload=14661]: Done uploading segments... Aug 7 23:55:22.799154191 (14h15m57.314256772s) [upload=14661]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:55:22.918485444 (14h15m57.433588022s) [upload=14677]: Starting upload Aug 7 23:55:22.918566621 (14h15m57.433672141s) [upload=14677]: Done starting upload Aug 7 23:55:22.918647605 (14h15m57.433750172s) [upload=14677]: Uploading segments... Aug 7 23:55:22.918705315 (14h15m57.4338079s) [upload=14677]: Got next segment. Inline: true Aug 7 23:55:22.918710403 (14h15m57.43381298s) [upload=14677]: Waiting for error group managing segments... Aug 7 23:55:22.918764176 (14h15m57.433866753s) [upload=14677]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:55:22.918769239 (14h15m57.433871815s) [upload=14677]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:55:22.918772352 (14h15m57.433874928s) [upload=14677]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:56:22.891103920 (14h16m57.406206502s) [upload=14677]: Done uploading segments... Aug 7 23:56:22.891142892 (14h16m57.406245471s) [upload=14677]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:56:23.012512156 (14h16m57.527614733s) [upload=14690]: Starting upload Aug 7 23:56:23.012570683 (14h16m57.527673265s) [upload=14690]: Done starting upload Aug 7 23:56:23.012632890 (14h16m57.527735455s) [upload=14690]: Uploading segments... Aug 7 23:56:23.012675514 (14h16m57.527778093s) [upload=14690]: Got next segment. Inline: true Aug 7 23:56:23.012680516 (14h16m57.527783092s) [upload=14690]: Waiting for error group managing segments... Aug 7 23:56:23.012734720 (14h16m57.527837295s) [upload=14690]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:56:23.012739356 (14h16m57.527841923s) [upload=14690]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:56:23.012742484 (14h16m57.527845059s) [upload=14690]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:57:22.981126119 (14h17m57.496228686s) [upload=14690]: Done uploading segments... Aug 7 23:57:22.981169317 (14h17m57.496271889s) [upload=14690]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:57:23.078169780 (14h17m57.593272348s) [upload=14705]: Starting upload Aug 7 23:57:23.078287503 (14h17m57.593390079s) [upload=14705]: Done starting upload Aug 7 23:57:23.078353530 (14h17m57.593456107s) [upload=14705]: Uploading segments... Aug 7 23:57:23.078394403 (14h17m57.593496992s) [upload=14705]: Got next segment. Inline: true Aug 7 23:57:23.078400888 (14h17m57.593503457s) [upload=14705]: Waiting for error group managing segments... Aug 7 23:57:23.078452630 (14h17m57.593555201s) [upload=14705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:57:23.078457270 (14h17m57.593559838s) [upload=14705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:57:23.078460483 (14h17m57.593563048s) [upload=14705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:58:23.049396934 (14h18m57.564499518s) [upload=14705]: Done uploading segments... Aug 7 23:58:23.049418059 (14h18m57.564520634s) [upload=14705]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:58:23.169720285 (14h18m57.684822852s) [upload=14722]: Starting upload Aug 7 23:58:23.169779212 (14h18m57.684881788s) [upload=14722]: Done starting upload Aug 7 23:58:23.169840834 (14h18m57.684943401s) [upload=14722]: Uploading segments... Aug 7 23:58:23.169880437 (14h18m57.684983021s) [upload=14722]: Got next segment. Inline: true Aug 7 23:58:23.169885772 (14h18m57.684988338s) [upload=14722]: Waiting for error group managing segments... Aug 7 23:58:23.169927728 (14h18m57.685030303s) [upload=14722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:58:23.169933919 (14h18m57.685036498s) [upload=14722]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:58:23.169938117 (14h18m57.685040704s) [upload=14722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 23:59:23.138140642 (14h19m57.653243209s) [upload=14722]: Done uploading segments... Aug 7 23:59:23.138159469 (14h19m57.653262035s) [upload=14722]: Upload finished. err: metaclient: context deadline exceeded Aug 7 23:59:35.502179618 (14h20m10.017282184s) [upload=14745]: Starting upload Aug 7 23:59:35.502246881 (14h20m10.017349449s) [upload=14745]: Done starting upload Aug 7 23:59:35.502308502 (14h20m10.017411071s) [upload=14745]: Uploading segments... Aug 7 23:59:35.502347394 (14h20m10.017449979s) [upload=14745]: Got next segment. Inline: true Aug 7 23:59:35.502351750 (14h20m10.017454327s) [upload=14745]: Waiting for error group managing segments... Aug 7 23:59:35.502390443 (14h20m10.017493009s) [upload=14745]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 23:59:35.502394821 (14h20m10.017497398s) [upload=14745]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 23:59:35.502398119 (14h20m10.017500695s) [upload=14745]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:35.472447782 (14h21m9.987550364s) [upload=14745]: Done uploading segments... Aug 8 00:00:35.472473966 (14h21m9.987576558s) [upload=14745]: Upload finished. err: metaclient: context deadline exceeded Aug 8 00:00:35.593243192 (14h21m10.10834576s) [upload=14760]: Starting upload Aug 8 00:00:35.593395835 (14h21m10.108498415s) [upload=14760]: Done starting upload Aug 8 00:00:35.593497926 (14h21m10.10860049s) [upload=14760]: Uploading segments... Aug 8 00:00:35.593659481 (14h21m10.108762045s) [upload=14760]: Got next segment. Inline: true Aug 8 00:00:35.593667686 (14h21m10.108770273s) [upload=14760]: Waiting for error group managing segments... Aug 8 00:00:35.593750345 (14h21m10.10885293s) [upload=14760]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:00:35.593773930 (14h21m10.108876508s) [upload=14760]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:00:35.593782409 (14h21m10.108884991s) [upload=14760]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:00:35.662793141 (14h21m10.177895709s) [upload=14760]: Done uploading segments... Aug 8 00:00:35.662815087 (14h21m10.177917658s) [upload=14760]: Upload finished. err: <nil> Aug 8 00:01:23.648690375 (14h21m58.163792959s) [upload=14775]: Starting upload Aug 8 00:01:23.648799550 (14h21m58.163902126s) [upload=14775]: Done starting upload Aug 8 00:01:23.648928369 (14h21m58.164030934s) [upload=14775]: Uploading segments... Aug 8 00:01:23.648986999 (14h21m58.164089578s) [upload=14775]: Got next segment. Inline: true Aug 8 00:01:23.648992059 (14h21m58.164094634s) [upload=14775]: Waiting for error group managing segments... Aug 8 00:01:23.649037148 (14h21m58.164139714s) [upload=14775]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 8 00:01:23.649041722 (14h21m58.164144298s) [upload=14775]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 8 00:01:23.649044950 (14h21m58.164147527s) [upload=14775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 8 00:02:23.618855090 (14h22m58.133957671s) [upload=14775]: Done uploading segments... Aug 8 00:02:23.618900564 (14h22m58.134003142s) [upload=14775]: Upload finished. err: metaclient: context deadline exceeded Duration: 215.723178ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.XXXXX.............................................X........] duration: 2.165394746s FAIL (2025-08-08 00:05:38 UTC) operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.165424078s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)