Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 9 out of 60 is failed (threshold=3) [..........................................XXXXXX.XXX........] duration: 44.714637722s trace: https://jaeger.dev.storj.io/trace/51eaf95d18c2a564 FAIL (2026-04-02 00:12:21 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:05:55.554575910 (637h5m53.151346936s) [upload=713440]: Starting upload Apr 2 00:05:55.554663664 (637h5m53.15143469s) [upload=713440]: Done starting upload Apr 2 00:05:55.554808188 (637h5m53.151579223s) [upload=713440]: Uploading segments... Apr 2 00:05:55.554899119 (637h5m53.151670152s) [upload=713440]: Got next segment. Inline: true Apr 2 00:05:55.554906374 (637h5m53.151677398s) [upload=713440]: Waiting for error group managing segments... Apr 2 00:05:55.554972654 (637h5m53.151743686s) [upload=713440]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:05:55.554979172 (637h5m53.151750202s) [upload=713440]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Apr 2 00:05:55.554984144 (637h5m53.151755174s) [upload=713440]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:06:50.731115863 (637h6m48.327886896s) [upload=713440]: Done uploading segments... Apr 2 00:06:50.731163927 (637h6m48.32793495s) [upload=713440]: Upload finished. err: <nil> Duration:* 44.71467841s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [........................................XXXXXXXXXXXXXXXXXXXX] duration: 5.018506866s trace: https://jaeger.dev.storj.io/trace/56e115691e53031b FAIL (2026-04-02 00:12:09 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.018784765s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 1.031199672s trace: https://jaeger.dev.storj.io/trace/3c28cdfe17d1a942 FAIL (2026-04-02 00:11:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:02:37.851070386 (637h2m35.44784141s) [upload=713386]: Starting upload Apr 2 00:02:37.851166355 (637h2m35.447937379s) [upload=713386]: Done starting upload Apr 2 00:02:37.851253090 (637h2m35.44802412s) [upload=713386]: Uploading segments... Apr 2 00:02:37.851793181 (637h2m35.448564196s) [upload=713386]: Got next segment. Inline: false Apr 2 00:02:37.851882102 (637h2m35.448653125s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:02:37.851889461 (637h2m35.448660484s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:02:37.880067208 (637h2m35.476838238s) [seg_pos={0 0}, upload=713386]: Begin upload segment... Apr 2 00:02:37.880168602 (637h2m35.476939634s) [seg_pos={0 0}, upload=713386]: Done begin upload segment. Apr 2 00:02:37.880207987 (637h2m35.476979021s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880214330 (637h2m35.476985339s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880292505 (637h2m35.477063531s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881027652 (637h2m35.477798688s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881378064 (637h2m35.478149089s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882097992 (637h2m35.478869026s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882586634 (637h2m35.479357658s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.883643537 (637h2m35.48041457s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884145002 (637h2m35.480916034s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884766674 (637h2m35.481537743s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885263298 (637h2m35.482034334s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885763219 (637h2m35.482534252s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886298227 (637h2m35.483069261s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886892344 (637h2m35.483663375s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.887518235 (637h2m35.484289271s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.888216955 (637h2m35.484987987s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.976004543 (637h2m35.572775567s) [upload=713386]: Next returned nil segment Apr 2 00:02:37.976025238 (637h2m35.572796273s) [upload=713386]: Waiting for error group managing segments... Apr 2 00:02:37.982357614 (637h2m35.579128639s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.982449885 (637h2m35.5792209s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983162285 (637h2m35.579933315s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983404595 (637h2m35.580175625s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983964188 (637h2m35.580735217s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.984025323 (637h2m35.58079634s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.985346526 (637h2m35.582117555s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986081391 (637h2m35.582852422s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986551623 (637h2m35.583322653s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.987396144 (637h2m35.584167177s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.997416565 (637h2m35.594187596s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005002081 (637h2m35.601773128s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005136838 (637h2m35.601907854s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005157484 (637h2m35.601928499s) [seg_pos={0 0}, upload=713386]: Satisfied cohort requirements after 13 pieces Apr 2 00:02:38.005386492 (637h2m35.602157525s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005396281 (637h2m35.60216731s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005531231 (637h2m35.602302263s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005570651 (637h2m35.602341684s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005692951 (637h2m35.602463983s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005700918 (637h2m35.602471942s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005722537 (637h2m35.60249356s) [seg_pos={0 0}, upload=713386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 2 00:02:38.005805596 (637h2m35.60257662s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:02:38.005812420 (637h2m35.602583444s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:03:07.848626111 (637h3m5.445397142s) [upload=713386]: Done uploading segments... Apr 2 00:03:07.848811089 (637h3m5.445582113s) [upload=713386]: Upload finished. err: metaclient: context deadline exceeded Duration: 1.031237166s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 9 out of 60 is failed (threshold=3) [...........................................XXXXXX.XXX.......] duration: 19.02774825s trace: https://jaeger.dev.storj.io/trace/51eaf95d18c2a564 FAIL (2026-04-02 00:11:11 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:05:55.554575910 (637h5m53.151346936s) [upload=713440]: Starting upload Apr 2 00:05:55.554663664 (637h5m53.15143469s) [upload=713440]: Done starting upload Apr 2 00:05:55.554808188 (637h5m53.151579223s) [upload=713440]: Uploading segments... Apr 2 00:05:55.554899119 (637h5m53.151670152s) [upload=713440]: Got next segment. Inline: true Apr 2 00:05:55.554906374 (637h5m53.151677398s) [upload=713440]: Waiting for error group managing segments... Apr 2 00:05:55.554972654 (637h5m53.151743686s) [upload=713440]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:05:55.554979172 (637h5m53.151750202s) [upload=713440]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Apr 2 00:05:55.554984144 (637h5m53.151755174s) [upload=713440]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:06:50.731115863 (637h6m48.327886896s) [upload=713440]: Done uploading segments... Apr 2 00:06:50.731163927 (637h6m48.32793495s) [upload=713440]: Upload finished. err: <nil> Duration:* 19.027805715s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.........................................XXXXXXXXXXXXXXXXXXX] duration: 5.001486635s trace: https://jaeger.dev.storj.io/trace/2771d5022205ae1c FAIL (2026-04-02 00:11:11 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001588551s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [................................................XX.XXX...X..] duration: 23.588341356s trace: https://jaeger.dev.storj.io/trace/7b66886c63182d02 FAIL (2026-04-02 00:11:00 UTC) operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:08:32.201995061 (637h8m29.798766091s) [upload=713490]: Starting upload Apr 2 00:08:32.202058979 (637h8m29.798830011s) [upload=713490]: Done starting upload Apr 2 00:08:32.202104136 (637h8m29.798875153s) [upload=713490]: Uploading segments... Apr 2 00:08:32.202468215 (637h8m29.799239227s) [upload=713490]: Got next segment. Inline: false Apr 2 00:08:32.202545730 (637h8m29.799316756s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:08:32.202554537 (637h8m29.799325554s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:08:32.236884387 (637h8m29.83365542s) [seg_pos={0 0}, upload=713490]: Begin upload segment... Apr 2 00:08:32.236972300 (637h8m29.833743327s) [seg_pos={0 0}, upload=713490]: Done begin upload segment. Apr 2 00:08:32.237040287 (637h8m29.833811319s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237609577 (637h8m29.834380609s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237761417 (637h8m29.834532443s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.238323459 (637h8m29.835094484s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239091520 (637h8m29.835862549s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239476832 (637h8m29.836247858s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240185307 (637h8m29.836956336s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240585259 (637h8m29.837356286s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241120411 (637h8m29.83789144s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241483943 (637h8m29.838254967s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241758194 (637h8m29.838529219s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.242223914 (637h8m29.838994946s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243144784 (637h8m29.839915821s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243610121 (637h8m29.840381134s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244322341 (637h8m29.841093371s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244793070 (637h8m29.841564104s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245167641 (637h8m29.841938667s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245545293 (637h8m29.842316322s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.309369659 (637h8m29.9061407s) [upload=713490]: Next returned nil segment Apr 2 00:08:32.309386955 (637h8m29.906157989s) [upload=713490]: Waiting for error group managing segments... Apr 2 00:08:32.314998992 (637h8m29.911770022s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.317608657 (637h8m29.914379694s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.331442646 (637h8m29.92821367s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.334324786 (637h8m29.931095819s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.347843004 (637h8m29.944614014s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.349783033 (637h8m29.946554099s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.449314364 (637h8m30.04608539s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.466602858 (637h8m30.063373882s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.620540338 (637h8m30.217311348s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.627266502 (637h8m30.224037516s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.633186725 (637h8m30.229957751s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.836095281 (637h8m30.432866306s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.841319166 (637h8m30.438090195s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852957995 (637h8m30.449729018s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852982095 (637h8m30.449753118s) [seg_pos={0 0}, upload=713490]: Satisfied cohort requirements after 14 pieces Apr 2 00:08:32.853223544 (637h8m30.449994576s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853232550 (637h8m30.450003573s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853611136 (637h8m30.450382165s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MVWRaM417kW63ffaSvNzgtk1PxPQbbvDAxEzFSLVjDkhkLt7f): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853619602 (637h8m30.450390625s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853821693 (637h8m30.450592723s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GihGm5Lo1EDs1ZMqctuqxN3VVpvZHz7iBo4aSvhRhF2sBkAKx): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853828723 (637h8m30.450599745s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853975694 (637h8m30.450746723s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853981928 (637h8m30.450752951s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.854004392 (637h8m30.450775415s) [seg_pos={0 0}, upload=713490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 2 00:08:32.854137807 (637h8m30.450908831s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:08:32.854144625 (637h8m30.450915648s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:08:32.919468065 (637h8m30.516239079s) [upload=713490]: Done uploading segments... Apr 2 00:08:32.919489033 (637h8m30.516260054s) [upload=713490]: Upload finished. err: <nil> Duration: 23.588385079s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.101656767s trace: https://jaeger.dev.storj.io/trace/3c28cdfe17d1a942 FAIL (2026-04-02 00:10:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:02:37.851070386 (637h2m35.44784141s) [upload=713386]: Starting upload Apr 2 00:02:37.851166355 (637h2m35.447937379s) [upload=713386]: Done starting upload Apr 2 00:02:37.851253090 (637h2m35.44802412s) [upload=713386]: Uploading segments... Apr 2 00:02:37.851793181 (637h2m35.448564196s) [upload=713386]: Got next segment. Inline: false Apr 2 00:02:37.851882102 (637h2m35.448653125s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:02:37.851889461 (637h2m35.448660484s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:02:37.880067208 (637h2m35.476838238s) [seg_pos={0 0}, upload=713386]: Begin upload segment... Apr 2 00:02:37.880168602 (637h2m35.476939634s) [seg_pos={0 0}, upload=713386]: Done begin upload segment. Apr 2 00:02:37.880207987 (637h2m35.476979021s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880214330 (637h2m35.476985339s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880292505 (637h2m35.477063531s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881027652 (637h2m35.477798688s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881378064 (637h2m35.478149089s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882097992 (637h2m35.478869026s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882586634 (637h2m35.479357658s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.883643537 (637h2m35.48041457s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884145002 (637h2m35.480916034s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884766674 (637h2m35.481537743s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885263298 (637h2m35.482034334s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885763219 (637h2m35.482534252s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886298227 (637h2m35.483069261s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886892344 (637h2m35.483663375s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.887518235 (637h2m35.484289271s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.888216955 (637h2m35.484987987s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.976004543 (637h2m35.572775567s) [upload=713386]: Next returned nil segment Apr 2 00:02:37.976025238 (637h2m35.572796273s) [upload=713386]: Waiting for error group managing segments... Apr 2 00:02:37.982357614 (637h2m35.579128639s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.982449885 (637h2m35.5792209s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983162285 (637h2m35.579933315s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983404595 (637h2m35.580175625s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983964188 (637h2m35.580735217s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.984025323 (637h2m35.58079634s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.985346526 (637h2m35.582117555s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986081391 (637h2m35.582852422s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986551623 (637h2m35.583322653s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.987396144 (637h2m35.584167177s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.997416565 (637h2m35.594187596s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005002081 (637h2m35.601773128s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005136838 (637h2m35.601907854s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005157484 (637h2m35.601928499s) [seg_pos={0 0}, upload=713386]: Satisfied cohort requirements after 13 pieces Apr 2 00:02:38.005386492 (637h2m35.602157525s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005396281 (637h2m35.60216731s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005531231 (637h2m35.602302263s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005570651 (637h2m35.602341684s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005692951 (637h2m35.602463983s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005700918 (637h2m35.602471942s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005722537 (637h2m35.60249356s) [seg_pos={0 0}, upload=713386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 2 00:02:38.005805596 (637h2m35.60257662s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:02:38.005812420 (637h2m35.602583444s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:03:07.848626111 (637h3m5.445397142s) [upload=713386]: Done uploading segments... Apr 2 00:03:07.848811089 (637h3m5.445582113s) [upload=713386]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.101674351s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 9 out of 60 is failed (threshold=3) [............................................XXXXXX.XXX......] duration: 14.449281357s trace: https://jaeger.dev.storj.io/trace/51eaf95d18c2a564 FAIL (2026-04-02 00:10:49 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:05:55.554575910 (637h5m53.151346936s) [upload=713440]: Starting upload Apr 2 00:05:55.554663664 (637h5m53.15143469s) [upload=713440]: Done starting upload Apr 2 00:05:55.554808188 (637h5m53.151579223s) [upload=713440]: Uploading segments... Apr 2 00:05:55.554899119 (637h5m53.151670152s) [upload=713440]: Got next segment. Inline: true Apr 2 00:05:55.554906374 (637h5m53.151677398s) [upload=713440]: Waiting for error group managing segments... Apr 2 00:05:55.554972654 (637h5m53.151743686s) [upload=713440]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:05:55.554979172 (637h5m53.151750202s) [upload=713440]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Apr 2 00:05:55.554984144 (637h5m53.151755174s) [upload=713440]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:06:50.731115863 (637h6m48.327886896s) [upload=713440]: Done uploading segments... Apr 2 00:06:50.731163927 (637h6m48.32793495s) [upload=713440]: Upload finished. err: <nil> Duration:* 14.449304459s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 9 out of 60 is failed (threshold=3) [.............................................XXXXXX.XXX.....] duration: 22.870784918s trace: https://jaeger.dev.storj.io/trace/51eaf95d18c2a564 FAIL (2026-04-02 00:10:35 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:05:55.554575910 (637h5m53.151346936s) [upload=713440]: Starting upload Apr 2 00:05:55.554663664 (637h5m53.15143469s) [upload=713440]: Done starting upload Apr 2 00:05:55.554808188 (637h5m53.151579223s) [upload=713440]: Uploading segments... Apr 2 00:05:55.554899119 (637h5m53.151670152s) [upload=713440]: Got next segment. Inline: true Apr 2 00:05:55.554906374 (637h5m53.151677398s) [upload=713440]: Waiting for error group managing segments... Apr 2 00:05:55.554972654 (637h5m53.151743686s) [upload=713440]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:05:55.554979172 (637h5m53.151750202s) [upload=713440]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Apr 2 00:05:55.554984144 (637h5m53.151755174s) [upload=713440]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:06:50.731115863 (637h6m48.327886896s) [upload=713440]: Done uploading segments... Apr 2 00:06:50.731163927 (637h6m48.32793495s) [upload=713440]: Upload finished. err: <nil> Duration:* 22.870806872s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [..........................................XXXXXXXXXXXXXXXXXX] duration: 5.000855016s trace: https://jaeger.dev.storj.io/trace/21bc105af9f89879 FAIL (2026-04-02 00:10:34 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000902386s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 9 out of 60 is failed (threshold=3) [..............................................XXXXXX.XXX....] duration: 36.395941858s trace: https://jaeger.dev.storj.io/trace/51eaf95d18c2a564 FAIL (2026-04-02 00:10:12 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:05:55.554575910 (637h5m53.151346936s) [upload=713440]: Starting upload Apr 2 00:05:55.554663664 (637h5m53.15143469s) [upload=713440]: Done starting upload Apr 2 00:05:55.554808188 (637h5m53.151579223s) [upload=713440]: Uploading segments... Apr 2 00:05:55.554899119 (637h5m53.151670152s) [upload=713440]: Got next segment. Inline: true Apr 2 00:05:55.554906374 (637h5m53.151677398s) [upload=713440]: Waiting for error group managing segments... Apr 2 00:05:55.554972654 (637h5m53.151743686s) [upload=713440]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:05:55.554979172 (637h5m53.151750202s) [upload=713440]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Apr 2 00:05:55.554984144 (637h5m53.151755174s) [upload=713440]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:06:50.731115863 (637h6m48.327886896s) [upload=713440]: Done uploading segments... Apr 2 00:06:50.731163927 (637h6m48.32793495s) [upload=713440]: Upload finished. err: <nil> Duration:* 36.396044231s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 925.628177ms trace: https://jaeger.dev.storj.io/trace/3c28cdfe17d1a942 FAIL (2026-04-02 00:10:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:02:37.851070386 (637h2m35.44784141s) [upload=713386]: Starting upload Apr 2 00:02:37.851166355 (637h2m35.447937379s) [upload=713386]: Done starting upload Apr 2 00:02:37.851253090 (637h2m35.44802412s) [upload=713386]: Uploading segments... Apr 2 00:02:37.851793181 (637h2m35.448564196s) [upload=713386]: Got next segment. Inline: false Apr 2 00:02:37.851882102 (637h2m35.448653125s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:02:37.851889461 (637h2m35.448660484s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:02:37.880067208 (637h2m35.476838238s) [seg_pos={0 0}, upload=713386]: Begin upload segment... Apr 2 00:02:37.880168602 (637h2m35.476939634s) [seg_pos={0 0}, upload=713386]: Done begin upload segment. Apr 2 00:02:37.880207987 (637h2m35.476979021s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880214330 (637h2m35.476985339s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880292505 (637h2m35.477063531s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881027652 (637h2m35.477798688s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881378064 (637h2m35.478149089s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882097992 (637h2m35.478869026s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882586634 (637h2m35.479357658s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.883643537 (637h2m35.48041457s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884145002 (637h2m35.480916034s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884766674 (637h2m35.481537743s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885263298 (637h2m35.482034334s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885763219 (637h2m35.482534252s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886298227 (637h2m35.483069261s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886892344 (637h2m35.483663375s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.887518235 (637h2m35.484289271s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.888216955 (637h2m35.484987987s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.976004543 (637h2m35.572775567s) [upload=713386]: Next returned nil segment Apr 2 00:02:37.976025238 (637h2m35.572796273s) [upload=713386]: Waiting for error group managing segments... Apr 2 00:02:37.982357614 (637h2m35.579128639s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.982449885 (637h2m35.5792209s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983162285 (637h2m35.579933315s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983404595 (637h2m35.580175625s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983964188 (637h2m35.580735217s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.984025323 (637h2m35.58079634s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.985346526 (637h2m35.582117555s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986081391 (637h2m35.582852422s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986551623 (637h2m35.583322653s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.987396144 (637h2m35.584167177s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.997416565 (637h2m35.594187596s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005002081 (637h2m35.601773128s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005136838 (637h2m35.601907854s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005157484 (637h2m35.601928499s) [seg_pos={0 0}, upload=713386]: Satisfied cohort requirements after 13 pieces Apr 2 00:02:38.005386492 (637h2m35.602157525s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005396281 (637h2m35.60216731s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005531231 (637h2m35.602302263s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005570651 (637h2m35.602341684s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005692951 (637h2m35.602463983s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005700918 (637h2m35.602471942s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005722537 (637h2m35.60249356s) [seg_pos={0 0}, upload=713386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 2 00:02:38.005805596 (637h2m35.60257662s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:02:38.005812420 (637h2m35.602583444s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:03:07.848626111 (637h3m5.445397142s) [upload=713386]: Done uploading segments... Apr 2 00:03:07.848811089 (637h3m5.445582113s) [upload=713386]: Upload finished. err: metaclient: context deadline exceeded Duration: 925.648247ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................................................XX.XXX...X.] duration: 21.473436504s trace: https://jaeger.dev.storj.io/trace/7b66886c63182d02 FAIL (2026-04-02 00:09:53 UTC) operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:08:32.201995061 (637h8m29.798766091s) [upload=713490]: Starting upload Apr 2 00:08:32.202058979 (637h8m29.798830011s) [upload=713490]: Done starting upload Apr 2 00:08:32.202104136 (637h8m29.798875153s) [upload=713490]: Uploading segments... Apr 2 00:08:32.202468215 (637h8m29.799239227s) [upload=713490]: Got next segment. Inline: false Apr 2 00:08:32.202545730 (637h8m29.799316756s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:08:32.202554537 (637h8m29.799325554s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:08:32.236884387 (637h8m29.83365542s) [seg_pos={0 0}, upload=713490]: Begin upload segment... Apr 2 00:08:32.236972300 (637h8m29.833743327s) [seg_pos={0 0}, upload=713490]: Done begin upload segment. Apr 2 00:08:32.237040287 (637h8m29.833811319s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237609577 (637h8m29.834380609s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237761417 (637h8m29.834532443s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.238323459 (637h8m29.835094484s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239091520 (637h8m29.835862549s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239476832 (637h8m29.836247858s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240185307 (637h8m29.836956336s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240585259 (637h8m29.837356286s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241120411 (637h8m29.83789144s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241483943 (637h8m29.838254967s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241758194 (637h8m29.838529219s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.242223914 (637h8m29.838994946s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243144784 (637h8m29.839915821s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243610121 (637h8m29.840381134s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244322341 (637h8m29.841093371s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244793070 (637h8m29.841564104s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245167641 (637h8m29.841938667s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245545293 (637h8m29.842316322s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.309369659 (637h8m29.9061407s) [upload=713490]: Next returned nil segment Apr 2 00:08:32.309386955 (637h8m29.906157989s) [upload=713490]: Waiting for error group managing segments... Apr 2 00:08:32.314998992 (637h8m29.911770022s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.317608657 (637h8m29.914379694s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.331442646 (637h8m29.92821367s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.334324786 (637h8m29.931095819s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.347843004 (637h8m29.944614014s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.349783033 (637h8m29.946554099s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.449314364 (637h8m30.04608539s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.466602858 (637h8m30.063373882s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.620540338 (637h8m30.217311348s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.627266502 (637h8m30.224037516s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.633186725 (637h8m30.229957751s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.836095281 (637h8m30.432866306s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.841319166 (637h8m30.438090195s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852957995 (637h8m30.449729018s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852982095 (637h8m30.449753118s) [seg_pos={0 0}, upload=713490]: Satisfied cohort requirements after 14 pieces Apr 2 00:08:32.853223544 (637h8m30.449994576s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853232550 (637h8m30.450003573s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853611136 (637h8m30.450382165s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MVWRaM417kW63ffaSvNzgtk1PxPQbbvDAxEzFSLVjDkhkLt7f): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853619602 (637h8m30.450390625s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853821693 (637h8m30.450592723s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GihGm5Lo1EDs1ZMqctuqxN3VVpvZHz7iBo4aSvhRhF2sBkAKx): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853828723 (637h8m30.450599745s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853975694 (637h8m30.450746723s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853981928 (637h8m30.450752951s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.854004392 (637h8m30.450775415s) [seg_pos={0 0}, upload=713490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 2 00:08:32.854137807 (637h8m30.450908831s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:08:32.854144625 (637h8m30.450915648s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:08:32.919468065 (637h8m30.516239079s) [upload=713490]: Done uploading segments... Apr 2 00:08:32.919489033 (637h8m30.516260054s) [upload=713490]: Upload finished. err: <nil> Duration: 21.47347499s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................................................XX.XXX...X] duration: 1m0.004460432s trace: https://jaeger.dev.storj.io/trace/7b66886c63182d02 FAIL (2026-04-02 00:09:32 UTC) operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:08:32.201995061 (637h8m29.798766091s) [upload=713490]: Starting upload Apr 2 00:08:32.202058979 (637h8m29.798830011s) [upload=713490]: Done starting upload Apr 2 00:08:32.202104136 (637h8m29.798875153s) [upload=713490]: Uploading segments... Apr 2 00:08:32.202468215 (637h8m29.799239227s) [upload=713490]: Got next segment. Inline: false Apr 2 00:08:32.202545730 (637h8m29.799316756s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:08:32.202554537 (637h8m29.799325554s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:08:32.236884387 (637h8m29.83365542s) [seg_pos={0 0}, upload=713490]: Begin upload segment... Apr 2 00:08:32.236972300 (637h8m29.833743327s) [seg_pos={0 0}, upload=713490]: Done begin upload segment. Apr 2 00:08:32.237040287 (637h8m29.833811319s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237609577 (637h8m29.834380609s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237761417 (637h8m29.834532443s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.238323459 (637h8m29.835094484s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239091520 (637h8m29.835862549s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239476832 (637h8m29.836247858s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240185307 (637h8m29.836956336s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240585259 (637h8m29.837356286s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241120411 (637h8m29.83789144s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241483943 (637h8m29.838254967s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241758194 (637h8m29.838529219s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.242223914 (637h8m29.838994946s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243144784 (637h8m29.839915821s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243610121 (637h8m29.840381134s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244322341 (637h8m29.841093371s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244793070 (637h8m29.841564104s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245167641 (637h8m29.841938667s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245545293 (637h8m29.842316322s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.309369659 (637h8m29.9061407s) [upload=713490]: Next returned nil segment Apr 2 00:08:32.309386955 (637h8m29.906157989s) [upload=713490]: Waiting for error group managing segments... Apr 2 00:08:32.314998992 (637h8m29.911770022s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.317608657 (637h8m29.914379694s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.331442646 (637h8m29.92821367s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.334324786 (637h8m29.931095819s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.347843004 (637h8m29.944614014s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.349783033 (637h8m29.946554099s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.449314364 (637h8m30.04608539s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.466602858 (637h8m30.063373882s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.620540338 (637h8m30.217311348s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.627266502 (637h8m30.224037516s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.633186725 (637h8m30.229957751s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.836095281 (637h8m30.432866306s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.841319166 (637h8m30.438090195s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852957995 (637h8m30.449729018s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852982095 (637h8m30.449753118s) [seg_pos={0 0}, upload=713490]: Satisfied cohort requirements after 14 pieces Apr 2 00:08:32.853223544 (637h8m30.449994576s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853232550 (637h8m30.450003573s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853611136 (637h8m30.450382165s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MVWRaM417kW63ffaSvNzgtk1PxPQbbvDAxEzFSLVjDkhkLt7f): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853619602 (637h8m30.450390625s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853821693 (637h8m30.450592723s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GihGm5Lo1EDs1ZMqctuqxN3VVpvZHz7iBo4aSvhRhF2sBkAKx): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853828723 (637h8m30.450599745s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853975694 (637h8m30.450746723s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853981928 (637h8m30.450752951s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.854004392 (637h8m30.450775415s) [seg_pos={0 0}, upload=713490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 2 00:08:32.854137807 (637h8m30.450908831s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:08:32.854144625 (637h8m30.450915648s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:08:32.919468065 (637h8m30.516239079s) [upload=713490]: Done uploading segments... Apr 2 00:08:32.919489033 (637h8m30.516260054s) [upload=713490]: Upload finished. err: <nil> Duration: 1m0.004475854s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [...........................................XXXXXXXXXXXXXXXXX] duration: 5.000743693s trace: https://jaeger.dev.storj.io/trace/46d2cb3f7937b462 FAIL (2026-04-02 00:09:22 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000765077s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 9 out of 60 is failed (threshold=3) [...............................................XXXXXX.XXX...] duration: 37.913727944s trace: https://jaeger.dev.storj.io/trace/51eaf95d18c2a564 FAIL (2026-04-02 00:09:05 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:05:55.554575910 (637h5m53.151346936s) [upload=713440]: Starting upload Apr 2 00:05:55.554663664 (637h5m53.15143469s) [upload=713440]: Done starting upload Apr 2 00:05:55.554808188 (637h5m53.151579223s) [upload=713440]: Uploading segments... Apr 2 00:05:55.554899119 (637h5m53.151670152s) [upload=713440]: Got next segment. Inline: true Apr 2 00:05:55.554906374 (637h5m53.151677398s) [upload=713440]: Waiting for error group managing segments... Apr 2 00:05:55.554972654 (637h5m53.151743686s) [upload=713440]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:05:55.554979172 (637h5m53.151750202s) [upload=713440]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Apr 2 00:05:55.554984144 (637h5m53.151755174s) [upload=713440]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:06:50.731115863 (637h6m48.327886896s) [upload=713440]: Done uploading segments... Apr 2 00:06:50.731163927 (637h6m48.32793495s) [upload=713440]: Upload finished. err: <nil> Duration:* 37.913760032s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 947.108895ms trace: https://jaeger.dev.storj.io/trace/3c28cdfe17d1a942 FAIL (2026-04-02 00:09:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:02:37.851070386 (637h2m35.44784141s) [upload=713386]: Starting upload Apr 2 00:02:37.851166355 (637h2m35.447937379s) [upload=713386]: Done starting upload Apr 2 00:02:37.851253090 (637h2m35.44802412s) [upload=713386]: Uploading segments... Apr 2 00:02:37.851793181 (637h2m35.448564196s) [upload=713386]: Got next segment. Inline: false Apr 2 00:02:37.851882102 (637h2m35.448653125s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:02:37.851889461 (637h2m35.448660484s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:02:37.880067208 (637h2m35.476838238s) [seg_pos={0 0}, upload=713386]: Begin upload segment... Apr 2 00:02:37.880168602 (637h2m35.476939634s) [seg_pos={0 0}, upload=713386]: Done begin upload segment. Apr 2 00:02:37.880207987 (637h2m35.476979021s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880214330 (637h2m35.476985339s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880292505 (637h2m35.477063531s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881027652 (637h2m35.477798688s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881378064 (637h2m35.478149089s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882097992 (637h2m35.478869026s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882586634 (637h2m35.479357658s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.883643537 (637h2m35.48041457s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884145002 (637h2m35.480916034s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884766674 (637h2m35.481537743s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885263298 (637h2m35.482034334s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885763219 (637h2m35.482534252s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886298227 (637h2m35.483069261s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886892344 (637h2m35.483663375s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.887518235 (637h2m35.484289271s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.888216955 (637h2m35.484987987s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.976004543 (637h2m35.572775567s) [upload=713386]: Next returned nil segment Apr 2 00:02:37.976025238 (637h2m35.572796273s) [upload=713386]: Waiting for error group managing segments... Apr 2 00:02:37.982357614 (637h2m35.579128639s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.982449885 (637h2m35.5792209s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983162285 (637h2m35.579933315s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983404595 (637h2m35.580175625s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983964188 (637h2m35.580735217s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.984025323 (637h2m35.58079634s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.985346526 (637h2m35.582117555s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986081391 (637h2m35.582852422s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986551623 (637h2m35.583322653s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.987396144 (637h2m35.584167177s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.997416565 (637h2m35.594187596s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005002081 (637h2m35.601773128s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005136838 (637h2m35.601907854s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005157484 (637h2m35.601928499s) [seg_pos={0 0}, upload=713386]: Satisfied cohort requirements after 13 pieces Apr 2 00:02:38.005386492 (637h2m35.602157525s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005396281 (637h2m35.60216731s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005531231 (637h2m35.602302263s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005570651 (637h2m35.602341684s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005692951 (637h2m35.602463983s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005700918 (637h2m35.602471942s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005722537 (637h2m35.60249356s) [seg_pos={0 0}, upload=713386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 2 00:02:38.005805596 (637h2m35.60257662s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:02:38.005812420 (637h2m35.602583444s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:03:07.848626111 (637h3m5.445397142s) [upload=713386]: Done uploading segments... Apr 2 00:03:07.848811089 (637h3m5.445582113s) [upload=713386]: Upload finished. err: metaclient: context deadline exceeded Duration: 947.217273ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [............................................XXXXXXXXXXXXXXXX] duration: 5.002222329s trace: https://jaeger.dev.storj.io/trace/461a9a083306ed82 FAIL (2026-04-02 00:08:53 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00224369s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................................................XX.XXX...] duration: 48.326877564s trace: https://jaeger.dev.storj.io/trace/4b65c7f00c3a8dfe FAIL (2026-04-02 00:08:32 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:04:22.752055593 (637h4m20.348826621s) [upload=713415]: Starting upload Apr 2 00:04:22.752142293 (637h4m20.348913314s) [upload=713415]: Done starting upload Apr 2 00:04:22.752373740 (637h4m20.349144754s) [upload=713415]: Uploading segments... Apr 2 00:04:22.752920200 (637h4m20.349691209s) [upload=713415]: Got next segment. Inline: false Apr 2 00:04:22.753003102 (637h4m20.34977413s) [upload=713415]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:04:22.753010562 (637h4m20.34978159s) [upload=713415]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:04:22.783143384 (637h4m20.379914414s) [seg_pos={0 0}, upload=713415]: Begin upload segment... Apr 2 00:04:22.783237995 (637h4m20.380009023s) [seg_pos={0 0}, upload=713415]: Done begin upload segment. Apr 2 00:04:22.783303903 (637h4m20.380074935s) [noise=true , address=145.79.227.36:4544 , piece_id=E3OMMGC5CKYJUOX5B35AYKGNSDGE4VCGFPOEUOTYPC2XRAVRXBXQ, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.783526119 (637h4m20.380297197s) [noise=true , address=45.91.116.9:4279 , piece_id=DI5LFK62FQAZ3V53XMXBV5BF5GQOXXQV2ERCZMLG6W4SFLWZJNLA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.784039849 (637h4m20.380810882s) [noise=true , address=77.90.31.18:4829 , piece_id=JSLFRG37EPS5CPLV2PZWGZNGMVD2M6FHKIH3UOEWI5NVQYVZG3OA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.784935349 (637h4m20.381706381s) [noise=true , address=45.91.116.11:4146 , piece_id=YIQM5Q5D5ATOMRCLKUS65R64V6AP2GRI2LDYGAJC7PUEL5X5R2VQ, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.785481593 (637h4m20.382252626s) [noise=true , address=145.79.227.50:4393 , piece_id=MWJG53BIMK7ZJVIDWCCFLERHNJXLDT362XVF5G5YDZRQWZYP7DVA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.786055921 (637h4m20.382826954s) [noise=true , address=145.79.227.35:4510 , piece_id=5SG3QJUTQK4DTCOM7V4WITK26PGOJDTYC73VH26DDGO66JCHUZRA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.786577360 (637h4m20.383348392s) [noise=true , address=77.90.31.21:4886 , piece_id=VO76QGDVAOKYXJSQRNG3ZDKCRSTGTJHNFAPFMJGBKVWAGGQYEMXQ, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.787104779 (637h4m20.383875806s) [noise=true , address=193.149.164.67:4023 , piece_id=R3CUHCRBGEDESHIPF4V3IDJTRIEYCKINLETW7QX2VGEGYDZSWUOQ, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.787642700 (637h4m20.384413724s) [noise=true , address=77.90.31.16:4752 , piece_id=3UUPXGPJFEOPJTUST426KIIRVNZS3EZNDNR6I62WU7EK2TTTGYDA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.788131879 (637h4m20.384902913s) [noise=true , address=45.91.116.14:4212 , piece_id=HOWBZJIMTRVXD7ZIBNHQYG4MVF24CQFZ6U5DNGG5QCAPESYE52GQ, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.788712943 (637h4m20.385483978s) [noise=true , address=193.149.164.69:4077 , piece_id=67SD52WWMRLAPWQUXDFYBH6NUN2GUIELOYDQAP276TIGJXI3NHXA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.790495292 (637h4m20.387266326s) [noise=true , address=193.149.164.70:4136 , piece_id=FNOIOGNFNTDV73LM3LPRUXBDGT344E5X6ILPDLE4NIE4EUILPT3A, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.790801985 (637h4m20.387573018s) [noise=true , address=64.49.13.137:4494 , piece_id=Y3ZLPGV6HDTQ6PTNOZL4XMDWA477P7IFILBILOUVRAIVNT2B4BAA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.793254870 (637h4m20.390025935s) [noise=true , address=167.17.72.70:4762 , piece_id=YNFEXS5LSOADFL7N5YRNS5SSN7O3CULTAJ7GV6L3AQ6QPBC7FHQA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.795129210 (637h4m20.391900248s) [noise=true , address=45.91.116.8:4059 , piece_id=RNXUSRZSYEH3U5FI67FXNMWV3UGRWWJRMMCRUFK4L4S3UCG7I65A, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.796559075 (637h4m20.393330105s) [noise=true , address=45.91.116.13:4175 , piece_id=5LJ7QONPGVNOV5RJBVREG6M7WJQ3OLAHP72LSAXBFIM3YJZ65DNQ, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.804221994 (637h4m20.400993226s) [noise=true , address=193.149.164.68:4038 , piece_id=WQQKZ4FCTWQOWBZMKU5THZFZE3FAGMSPR7LVMA6RCTHQXLLLGFHA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.808771177 (637h4m20.405542202s) [noise=true , address=145.79.227.5:4238 , piece_id=WUJT6YJXY6Z6GUJEV5LJ6IBNFZCPOHNDKTWXTN4B5KJYSYBU2XEA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.859136944 (637h4m20.455907956s) [upload=713415]: Next returned nil segment Apr 2 00:04:22.859154635 (637h4m20.45592567s) [upload=713415]: Waiting for error group managing segments... Apr 2 00:04:22.864190905 (637h4m20.460961929s) [noise=true , address=45.91.116.8:4059 , piece_id=RNXUSRZSYEH3U5FI67FXNMWV3UGRWWJRMMCRUFK4L4S3UCG7I65A, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.865233224 (637h4m20.462004248s) [noise=true , address=145.79.227.5:4238 , piece_id=WUJT6YJXY6Z6GUJEV5LJ6IBNFZCPOHNDKTWXTN4B5KJYSYBU2XEA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.865328660 (637h4m20.462099686s) [noise=true , address=45.91.116.9:4279 , piece_id=DI5LFK62FQAZ3V53XMXBV5BF5GQOXXQV2ERCZMLG6W4SFLWZJNLA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.865956793 (637h4m20.462727822s) [noise=true , address=77.90.31.16:4752 , piece_id=3UUPXGPJFEOPJTUST426KIIRVNZS3EZNDNR6I62WU7EK2TTTGYDA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.868160098 (637h4m20.464931123s) [noise=true , address=145.79.227.35:4510 , piece_id=5SG3QJUTQK4DTCOM7V4WITK26PGOJDTYC73VH26DDGO66JCHUZRA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.868583222 (637h4m20.465354253s) [noise=true , address=45.91.116.13:4175 , piece_id=5LJ7QONPGVNOV5RJBVREG6M7WJQ3OLAHP72LSAXBFIM3YJZ65DNQ, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.877818662 (637h4m20.47458969s) [noise=true , address=145.79.227.36:4544 , piece_id=E3OMMGC5CKYJUOX5B35AYKGNSDGE4VCGFPOEUOTYPC2XRAVRXBXQ, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.878065899 (637h4m20.474836927s) [noise=true , address=77.90.31.18:4829 , piece_id=JSLFRG37EPS5CPLV2PZWGZNGMVD2M6FHKIH3UOEWI5NVQYVZG3OA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.878562340 (637h4m20.475333372s) [noise=true , address=45.91.116.14:4212 , piece_id=HOWBZJIMTRVXD7ZIBNHQYG4MVF24CQFZ6U5DNGG5QCAPESYE52GQ, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.879067238 (637h4m20.475838265s) [noise=true , address=77.90.31.21:4886 , piece_id=VO76QGDVAOKYXJSQRNG3ZDKCRSTGTJHNFAPFMJGBKVWAGGQYEMXQ, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.880940501 (637h4m20.477711531s) [noise=true , address=145.79.227.50:4393 , piece_id=MWJG53BIMK7ZJVIDWCCFLERHNJXLDT362XVF5G5YDZRQWZYP7DVA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.881924153 (637h4m20.478695178s) [noise=true , address=45.91.116.11:4146 , piece_id=YIQM5Q5D5ATOMRCLKUS65R64V6AP2GRI2LDYGAJC7PUEL5X5R2VQ, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:23.182938845 (637h4m20.779709859s) [noise=true , address=167.17.72.70:4762 , piece_id=YNFEXS5LSOADFL7N5YRNS5SSN7O3CULTAJ7GV6L3AQ6QPBC7FHQA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:23.389241087 (637h4m20.98601211s) [noise=true , address=193.149.164.67:4023 , piece_id=R3CUHCRBGEDESHIPF4V3IDJTRIEYCKINLETW7QX2VGEGYDZSWUOQ, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:23.389433377 (637h4m20.986204399s) [seg_pos={0 0}, upload=713415]: Satisfied cohort requirements after 14 pieces Apr 2 00:04:23.389804584 (637h4m20.98657561s) [noise=true , address=193.149.164.68:4038 , piece_id=WQQKZ4FCTWQOWBZMKU5THZFZE3FAGMSPR7LVMA6RCTHQXLLLGFHA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LvCkhgeYiXJLJtJSXmbrmWXxMCUTePAT6Ww6zaCbP2JyUhK6k): protocol: expected piece hash; context canceled; EOF Apr 2 00:04:23.389825318 (637h4m20.986596345s) [noise=true , address=193.149.164.68:4038 , piece_id=WQQKZ4FCTWQOWBZMKU5THZFZE3FAGMSPR7LVMA6RCTHQXLLLGFHA, seg_pos={0 0}, upload=713415]: Long tail cancellation due to optimal threshold Apr 2 00:04:23.390075793 (637h4m20.986846826s) [noise=true , address=64.49.13.137:4494 , piece_id=Y3ZLPGV6HDTQ6PTNOZL4XMDWA477P7IFILBILOUVRAIVNT2B4BAA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TX66fDVv3qu4yBGx9FNZTiArNeUnqxjMYwb3VGqYCQRe8RwoU): protocol: expected piece hash; context canceled; EOF Apr 2 00:04:23.390082975 (637h4m20.986853998s) [noise=true , address=64.49.13.137:4494 , piece_id=Y3ZLPGV6HDTQ6PTNOZL4XMDWA477P7IFILBILOUVRAIVNT2B4BAA, seg_pos={0 0}, upload=713415]: Long tail cancellation due to optimal threshold Apr 2 00:04:23.390452875 (637h4m20.98722391s) [noise=true , address=193.149.164.69:4077 , piece_id=67SD52WWMRLAPWQUXDFYBH6NUN2GUIELOYDQAP276TIGJXI3NHXA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qcwvrgkkT6Q8pVEuUu7fA4wwHXNidvCNnKhrU3DpdqAHKGtLG): protocol: expected piece hash; context canceled; EOF Apr 2 00:04:23.390466752 (637h4m20.987237821s) [noise=true , address=193.149.164.69:4077 , piece_id=67SD52WWMRLAPWQUXDFYBH6NUN2GUIELOYDQAP276TIGJXI3NHXA, seg_pos={0 0}, upload=713415]: Long tail cancellation due to optimal threshold Apr 2 00:04:23.390792198 (637h4m20.987563321s) [noise=true , address=193.149.164.70:4136 , piece_id=FNOIOGNFNTDV73LM3LPRUXBDGT344E5X6ILPDLE4NIE4EUILPT3A, seg_pos={0 0}, upload=713415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YVyrEWNRczm6PHNwrAp58CdCuZPxSSHpzAqZ3F914MqvUNjuw): protocol: expected piece hash; context canceled; EOF Apr 2 00:04:23.390806838 (637h4m20.987577907s) [noise=true , address=193.149.164.70:4136 , piece_id=FNOIOGNFNTDV73LM3LPRUXBDGT344E5X6ILPDLE4NIE4EUILPT3A, seg_pos={0 0}, upload=713415]: Long tail cancellation due to optimal threshold Apr 2 00:04:23.390851162 (637h4m20.987622206s) [seg_pos={0 0}, upload=713415]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 2 00:04:23.390967742 (637h4m20.987738776s) [upload=713415]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:04:23.390974630 (637h4m20.987745661s) [upload=713415]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:05:33.093755528 (637h5m30.690526555s) [upload=713415]: Done uploading segments... Apr 2 00:05:33.093876891 (637h5m30.69064792s) [upload=713415]: Upload finished. err: metaclient: context deadline exceeded Duration: 48.32690538s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 9 out of 60 is failed (threshold=3) [................................................XXXXXX.XXX..] duration: 40.890982643s trace: https://jaeger.dev.storj.io/trace/51eaf95d18c2a564 FAIL (2026-04-02 00:08:22 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:05:55.554575910 (637h5m53.151346936s) [upload=713440]: Starting upload Apr 2 00:05:55.554663664 (637h5m53.15143469s) [upload=713440]: Done starting upload Apr 2 00:05:55.554808188 (637h5m53.151579223s) [upload=713440]: Uploading segments... Apr 2 00:05:55.554899119 (637h5m53.151670152s) [upload=713440]: Got next segment. Inline: true Apr 2 00:05:55.554906374 (637h5m53.151677398s) [upload=713440]: Waiting for error group managing segments... Apr 2 00:05:55.554972654 (637h5m53.151743686s) [upload=713440]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:05:55.554979172 (637h5m53.151750202s) [upload=713440]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Apr 2 00:05:55.554984144 (637h5m53.151755174s) [upload=713440]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:06:50.731115863 (637h6m48.327886896s) [upload=713440]: Done uploading segments... Apr 2 00:06:50.731163927 (637h6m48.32793495s) [upload=713440]: Upload finished. err: <nil> Duration:* 40.890999712s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [.............................................XXXXXXXXXXXXXXX] duration: 5.000735907s trace: https://jaeger.dev.storj.io/trace/4ce89c6852997dfe FAIL (2026-04-02 00:08:19 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00076025s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.02056792s trace: https://jaeger.dev.storj.io/trace/3c28cdfe17d1a942 FAIL (2026-04-02 00:07:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:02:37.851070386 (637h2m35.44784141s) [upload=713386]: Starting upload Apr 2 00:02:37.851166355 (637h2m35.447937379s) [upload=713386]: Done starting upload Apr 2 00:02:37.851253090 (637h2m35.44802412s) [upload=713386]: Uploading segments... Apr 2 00:02:37.851793181 (637h2m35.448564196s) [upload=713386]: Got next segment. Inline: false Apr 2 00:02:37.851882102 (637h2m35.448653125s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:02:37.851889461 (637h2m35.448660484s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:02:37.880067208 (637h2m35.476838238s) [seg_pos={0 0}, upload=713386]: Begin upload segment... Apr 2 00:02:37.880168602 (637h2m35.476939634s) [seg_pos={0 0}, upload=713386]: Done begin upload segment. Apr 2 00:02:37.880207987 (637h2m35.476979021s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880214330 (637h2m35.476985339s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880292505 (637h2m35.477063531s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881027652 (637h2m35.477798688s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881378064 (637h2m35.478149089s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882097992 (637h2m35.478869026s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882586634 (637h2m35.479357658s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.883643537 (637h2m35.48041457s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884145002 (637h2m35.480916034s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884766674 (637h2m35.481537743s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885263298 (637h2m35.482034334s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885763219 (637h2m35.482534252s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886298227 (637h2m35.483069261s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886892344 (637h2m35.483663375s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.887518235 (637h2m35.484289271s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.888216955 (637h2m35.484987987s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.976004543 (637h2m35.572775567s) [upload=713386]: Next returned nil segment Apr 2 00:02:37.976025238 (637h2m35.572796273s) [upload=713386]: Waiting for error group managing segments... Apr 2 00:02:37.982357614 (637h2m35.579128639s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.982449885 (637h2m35.5792209s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983162285 (637h2m35.579933315s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983404595 (637h2m35.580175625s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983964188 (637h2m35.580735217s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.984025323 (637h2m35.58079634s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.985346526 (637h2m35.582117555s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986081391 (637h2m35.582852422s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986551623 (637h2m35.583322653s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.987396144 (637h2m35.584167177s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.997416565 (637h2m35.594187596s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005002081 (637h2m35.601773128s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005136838 (637h2m35.601907854s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005157484 (637h2m35.601928499s) [seg_pos={0 0}, upload=713386]: Satisfied cohort requirements after 13 pieces Apr 2 00:02:38.005386492 (637h2m35.602157525s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005396281 (637h2m35.60216731s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005531231 (637h2m35.602302263s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005570651 (637h2m35.602341684s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005692951 (637h2m35.602463983s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005700918 (637h2m35.602471942s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005722537 (637h2m35.60249356s) [seg_pos={0 0}, upload=713386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 2 00:02:38.005805596 (637h2m35.60257662s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:02:38.005812420 (637h2m35.602583444s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:03:07.848626111 (637h3m5.445397142s) [upload=713386]: Done uploading segments... Apr 2 00:03:07.848811089 (637h3m5.445582113s) [upload=713386]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.020600719s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 9 out of 60 is failed (threshold=3) [.................................................XXXXXX.XXX.] duration: 46.2326106s trace: https://jaeger.dev.storj.io/trace/51eaf95d18c2a564 FAIL (2026-04-02 00:07:41 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:05:55.554575910 (637h5m53.151346936s) [upload=713440]: Starting upload Apr 2 00:05:55.554663664 (637h5m53.15143469s) [upload=713440]: Done starting upload Apr 2 00:05:55.554808188 (637h5m53.151579223s) [upload=713440]: Uploading segments... Apr 2 00:05:55.554899119 (637h5m53.151670152s) [upload=713440]: Got next segment. Inline: true Apr 2 00:05:55.554906374 (637h5m53.151677398s) [upload=713440]: Waiting for error group managing segments... Apr 2 00:05:55.554972654 (637h5m53.151743686s) [upload=713440]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:05:55.554979172 (637h5m53.151750202s) [upload=713440]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Apr 2 00:05:55.554984144 (637h5m53.151755174s) [upload=713440]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:06:50.731115863 (637h6m48.327886896s) [upload=713440]: Done uploading segments... Apr 2 00:06:50.731163927 (637h6m48.32793495s) [upload=713440]: Upload finished. err: <nil> Duration:* 46.232678948s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 475.599359ms trace: https://jaeger.dev.storj.io/trace/3c28cdfe17d1a942 FAIL (2026-04-02 00:07:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:02:37.851070386 (637h2m35.44784141s) [upload=713386]: Starting upload Apr 2 00:02:37.851166355 (637h2m35.447937379s) [upload=713386]: Done starting upload Apr 2 00:02:37.851253090 (637h2m35.44802412s) [upload=713386]: Uploading segments... Apr 2 00:02:37.851793181 (637h2m35.448564196s) [upload=713386]: Got next segment. Inline: false Apr 2 00:02:37.851882102 (637h2m35.448653125s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:02:37.851889461 (637h2m35.448660484s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:02:37.880067208 (637h2m35.476838238s) [seg_pos={0 0}, upload=713386]: Begin upload segment... Apr 2 00:02:37.880168602 (637h2m35.476939634s) [seg_pos={0 0}, upload=713386]: Done begin upload segment. Apr 2 00:02:37.880207987 (637h2m35.476979021s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880214330 (637h2m35.476985339s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880292505 (637h2m35.477063531s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881027652 (637h2m35.477798688s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881378064 (637h2m35.478149089s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882097992 (637h2m35.478869026s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882586634 (637h2m35.479357658s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.883643537 (637h2m35.48041457s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884145002 (637h2m35.480916034s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884766674 (637h2m35.481537743s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885263298 (637h2m35.482034334s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885763219 (637h2m35.482534252s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886298227 (637h2m35.483069261s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886892344 (637h2m35.483663375s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.887518235 (637h2m35.484289271s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.888216955 (637h2m35.484987987s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.976004543 (637h2m35.572775567s) [upload=713386]: Next returned nil segment Apr 2 00:02:37.976025238 (637h2m35.572796273s) [upload=713386]: Waiting for error group managing segments... Apr 2 00:02:37.982357614 (637h2m35.579128639s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.982449885 (637h2m35.5792209s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983162285 (637h2m35.579933315s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983404595 (637h2m35.580175625s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983964188 (637h2m35.580735217s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.984025323 (637h2m35.58079634s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.985346526 (637h2m35.582117555s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986081391 (637h2m35.582852422s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986551623 (637h2m35.583322653s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.987396144 (637h2m35.584167177s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.997416565 (637h2m35.594187596s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005002081 (637h2m35.601773128s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005136838 (637h2m35.601907854s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005157484 (637h2m35.601928499s) [seg_pos={0 0}, upload=713386]: Satisfied cohort requirements after 13 pieces Apr 2 00:02:38.005386492 (637h2m35.602157525s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005396281 (637h2m35.60216731s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005531231 (637h2m35.602302263s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005570651 (637h2m35.602341684s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005692951 (637h2m35.602463983s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005700918 (637h2m35.602471942s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005722537 (637h2m35.60249356s) [seg_pos={0 0}, upload=713386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 2 00:02:38.005805596 (637h2m35.60257662s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:02:38.005812420 (637h2m35.602583444s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:03:07.848626111 (637h3m5.445397142s) [upload=713386]: Done uploading segments... Apr 2 00:03:07.848811089 (637h3m5.445582113s) [upload=713386]: Upload finished. err: metaclient: context deadline exceeded Duration: 475.629602ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [..............................................XXXXXXXXXXXXXX] duration: 5.00172103s trace: https://jaeger.dev.storj.io/trace/5146e56d38b050fa FAIL (2026-04-02 00:07:16 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001821622s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................................................XX.XXX..] duration: 36.169087285s trace: https://jaeger.dev.storj.io/trace/4b65c7f00c3a8dfe FAIL (2026-04-02 00:07:09 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:04:22.752055593 (637h4m20.348826621s) [upload=713415]: Starting upload Apr 2 00:04:22.752142293 (637h4m20.348913314s) [upload=713415]: Done starting upload Apr 2 00:04:22.752373740 (637h4m20.349144754s) [upload=713415]: Uploading segments... Apr 2 00:04:22.752920200 (637h4m20.349691209s) [upload=713415]: Got next segment. Inline: false Apr 2 00:04:22.753003102 (637h4m20.34977413s) [upload=713415]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:04:22.753010562 (637h4m20.34978159s) [upload=713415]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:04:22.783143384 (637h4m20.379914414s) [seg_pos={0 0}, upload=713415]: Begin upload segment... Apr 2 00:04:22.783237995 (637h4m20.380009023s) [seg_pos={0 0}, upload=713415]: Done begin upload segment. Apr 2 00:04:22.783303903 (637h4m20.380074935s) [noise=true , address=145.79.227.36:4544 , piece_id=E3OMMGC5CKYJUOX5B35AYKGNSDGE4VCGFPOEUOTYPC2XRAVRXBXQ, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.783526119 (637h4m20.380297197s) [noise=true , address=45.91.116.9:4279 , piece_id=DI5LFK62FQAZ3V53XMXBV5BF5GQOXXQV2ERCZMLG6W4SFLWZJNLA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.784039849 (637h4m20.380810882s) [noise=true , address=77.90.31.18:4829 , piece_id=JSLFRG37EPS5CPLV2PZWGZNGMVD2M6FHKIH3UOEWI5NVQYVZG3OA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.784935349 (637h4m20.381706381s) [noise=true , address=45.91.116.11:4146 , piece_id=YIQM5Q5D5ATOMRCLKUS65R64V6AP2GRI2LDYGAJC7PUEL5X5R2VQ, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.785481593 (637h4m20.382252626s) [noise=true , address=145.79.227.50:4393 , piece_id=MWJG53BIMK7ZJVIDWCCFLERHNJXLDT362XVF5G5YDZRQWZYP7DVA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.786055921 (637h4m20.382826954s) [noise=true , address=145.79.227.35:4510 , piece_id=5SG3QJUTQK4DTCOM7V4WITK26PGOJDTYC73VH26DDGO66JCHUZRA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.786577360 (637h4m20.383348392s) [noise=true , address=77.90.31.21:4886 , piece_id=VO76QGDVAOKYXJSQRNG3ZDKCRSTGTJHNFAPFMJGBKVWAGGQYEMXQ, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.787104779 (637h4m20.383875806s) [noise=true , address=193.149.164.67:4023 , piece_id=R3CUHCRBGEDESHIPF4V3IDJTRIEYCKINLETW7QX2VGEGYDZSWUOQ, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.787642700 (637h4m20.384413724s) [noise=true , address=77.90.31.16:4752 , piece_id=3UUPXGPJFEOPJTUST426KIIRVNZS3EZNDNR6I62WU7EK2TTTGYDA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.788131879 (637h4m20.384902913s) [noise=true , address=45.91.116.14:4212 , piece_id=HOWBZJIMTRVXD7ZIBNHQYG4MVF24CQFZ6U5DNGG5QCAPESYE52GQ, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.788712943 (637h4m20.385483978s) [noise=true , address=193.149.164.69:4077 , piece_id=67SD52WWMRLAPWQUXDFYBH6NUN2GUIELOYDQAP276TIGJXI3NHXA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.790495292 (637h4m20.387266326s) [noise=true , address=193.149.164.70:4136 , piece_id=FNOIOGNFNTDV73LM3LPRUXBDGT344E5X6ILPDLE4NIE4EUILPT3A, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.790801985 (637h4m20.387573018s) [noise=true , address=64.49.13.137:4494 , piece_id=Y3ZLPGV6HDTQ6PTNOZL4XMDWA477P7IFILBILOUVRAIVNT2B4BAA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.793254870 (637h4m20.390025935s) [noise=true , address=167.17.72.70:4762 , piece_id=YNFEXS5LSOADFL7N5YRNS5SSN7O3CULTAJ7GV6L3AQ6QPBC7FHQA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.795129210 (637h4m20.391900248s) [noise=true , address=45.91.116.8:4059 , piece_id=RNXUSRZSYEH3U5FI67FXNMWV3UGRWWJRMMCRUFK4L4S3UCG7I65A, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.796559075 (637h4m20.393330105s) [noise=true , address=45.91.116.13:4175 , piece_id=5LJ7QONPGVNOV5RJBVREG6M7WJQ3OLAHP72LSAXBFIM3YJZ65DNQ, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.804221994 (637h4m20.400993226s) [noise=true , address=193.149.164.68:4038 , piece_id=WQQKZ4FCTWQOWBZMKU5THZFZE3FAGMSPR7LVMA6RCTHQXLLLGFHA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.808771177 (637h4m20.405542202s) [noise=true , address=145.79.227.5:4238 , piece_id=WUJT6YJXY6Z6GUJEV5LJ6IBNFZCPOHNDKTWXTN4B5KJYSYBU2XEA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.859136944 (637h4m20.455907956s) [upload=713415]: Next returned nil segment Apr 2 00:04:22.859154635 (637h4m20.45592567s) [upload=713415]: Waiting for error group managing segments... Apr 2 00:04:22.864190905 (637h4m20.460961929s) [noise=true , address=45.91.116.8:4059 , piece_id=RNXUSRZSYEH3U5FI67FXNMWV3UGRWWJRMMCRUFK4L4S3UCG7I65A, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.865233224 (637h4m20.462004248s) [noise=true , address=145.79.227.5:4238 , piece_id=WUJT6YJXY6Z6GUJEV5LJ6IBNFZCPOHNDKTWXTN4B5KJYSYBU2XEA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.865328660 (637h4m20.462099686s) [noise=true , address=45.91.116.9:4279 , piece_id=DI5LFK62FQAZ3V53XMXBV5BF5GQOXXQV2ERCZMLG6W4SFLWZJNLA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.865956793 (637h4m20.462727822s) [noise=true , address=77.90.31.16:4752 , piece_id=3UUPXGPJFEOPJTUST426KIIRVNZS3EZNDNR6I62WU7EK2TTTGYDA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.868160098 (637h4m20.464931123s) [noise=true , address=145.79.227.35:4510 , piece_id=5SG3QJUTQK4DTCOM7V4WITK26PGOJDTYC73VH26DDGO66JCHUZRA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.868583222 (637h4m20.465354253s) [noise=true , address=45.91.116.13:4175 , piece_id=5LJ7QONPGVNOV5RJBVREG6M7WJQ3OLAHP72LSAXBFIM3YJZ65DNQ, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.877818662 (637h4m20.47458969s) [noise=true , address=145.79.227.36:4544 , piece_id=E3OMMGC5CKYJUOX5B35AYKGNSDGE4VCGFPOEUOTYPC2XRAVRXBXQ, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.878065899 (637h4m20.474836927s) [noise=true , address=77.90.31.18:4829 , piece_id=JSLFRG37EPS5CPLV2PZWGZNGMVD2M6FHKIH3UOEWI5NVQYVZG3OA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.878562340 (637h4m20.475333372s) [noise=true , address=45.91.116.14:4212 , piece_id=HOWBZJIMTRVXD7ZIBNHQYG4MVF24CQFZ6U5DNGG5QCAPESYE52GQ, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.879067238 (637h4m20.475838265s) [noise=true , address=77.90.31.21:4886 , piece_id=VO76QGDVAOKYXJSQRNG3ZDKCRSTGTJHNFAPFMJGBKVWAGGQYEMXQ, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.880940501 (637h4m20.477711531s) [noise=true , address=145.79.227.50:4393 , piece_id=MWJG53BIMK7ZJVIDWCCFLERHNJXLDT362XVF5G5YDZRQWZYP7DVA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.881924153 (637h4m20.478695178s) [noise=true , address=45.91.116.11:4146 , piece_id=YIQM5Q5D5ATOMRCLKUS65R64V6AP2GRI2LDYGAJC7PUEL5X5R2VQ, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:23.182938845 (637h4m20.779709859s) [noise=true , address=167.17.72.70:4762 , piece_id=YNFEXS5LSOADFL7N5YRNS5SSN7O3CULTAJ7GV6L3AQ6QPBC7FHQA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:23.389241087 (637h4m20.98601211s) [noise=true , address=193.149.164.67:4023 , piece_id=R3CUHCRBGEDESHIPF4V3IDJTRIEYCKINLETW7QX2VGEGYDZSWUOQ, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:23.389433377 (637h4m20.986204399s) [seg_pos={0 0}, upload=713415]: Satisfied cohort requirements after 14 pieces Apr 2 00:04:23.389804584 (637h4m20.98657561s) [noise=true , address=193.149.164.68:4038 , piece_id=WQQKZ4FCTWQOWBZMKU5THZFZE3FAGMSPR7LVMA6RCTHQXLLLGFHA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LvCkhgeYiXJLJtJSXmbrmWXxMCUTePAT6Ww6zaCbP2JyUhK6k): protocol: expected piece hash; context canceled; EOF Apr 2 00:04:23.389825318 (637h4m20.986596345s) [noise=true , address=193.149.164.68:4038 , piece_id=WQQKZ4FCTWQOWBZMKU5THZFZE3FAGMSPR7LVMA6RCTHQXLLLGFHA, seg_pos={0 0}, upload=713415]: Long tail cancellation due to optimal threshold Apr 2 00:04:23.390075793 (637h4m20.986846826s) [noise=true , address=64.49.13.137:4494 , piece_id=Y3ZLPGV6HDTQ6PTNOZL4XMDWA477P7IFILBILOUVRAIVNT2B4BAA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TX66fDVv3qu4yBGx9FNZTiArNeUnqxjMYwb3VGqYCQRe8RwoU): protocol: expected piece hash; context canceled; EOF Apr 2 00:04:23.390082975 (637h4m20.986853998s) [noise=true , address=64.49.13.137:4494 , piece_id=Y3ZLPGV6HDTQ6PTNOZL4XMDWA477P7IFILBILOUVRAIVNT2B4BAA, seg_pos={0 0}, upload=713415]: Long tail cancellation due to optimal threshold Apr 2 00:04:23.390452875 (637h4m20.98722391s) [noise=true , address=193.149.164.69:4077 , piece_id=67SD52WWMRLAPWQUXDFYBH6NUN2GUIELOYDQAP276TIGJXI3NHXA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qcwvrgkkT6Q8pVEuUu7fA4wwHXNidvCNnKhrU3DpdqAHKGtLG): protocol: expected piece hash; context canceled; EOF Apr 2 00:04:23.390466752 (637h4m20.987237821s) [noise=true , address=193.149.164.69:4077 , piece_id=67SD52WWMRLAPWQUXDFYBH6NUN2GUIELOYDQAP276TIGJXI3NHXA, seg_pos={0 0}, upload=713415]: Long tail cancellation due to optimal threshold Apr 2 00:04:23.390792198 (637h4m20.987563321s) [noise=true , address=193.149.164.70:4136 , piece_id=FNOIOGNFNTDV73LM3LPRUXBDGT344E5X6ILPDLE4NIE4EUILPT3A, seg_pos={0 0}, upload=713415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YVyrEWNRczm6PHNwrAp58CdCuZPxSSHpzAqZ3F914MqvUNjuw): protocol: expected piece hash; context canceled; EOF Apr 2 00:04:23.390806838 (637h4m20.987577907s) [noise=true , address=193.149.164.70:4136 , piece_id=FNOIOGNFNTDV73LM3LPRUXBDGT344E5X6ILPDLE4NIE4EUILPT3A, seg_pos={0 0}, upload=713415]: Long tail cancellation due to optimal threshold Apr 2 00:04:23.390851162 (637h4m20.987622206s) [seg_pos={0 0}, upload=713415]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 2 00:04:23.390967742 (637h4m20.987738776s) [upload=713415]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:04:23.390974630 (637h4m20.987745661s) [upload=713415]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:05:33.093755528 (637h5m30.690526555s) [upload=713415]: Done uploading segments... Apr 2 00:05:33.093876891 (637h5m30.69064792s) [upload=713415]: Upload finished. err: metaclient: context deadline exceeded Duration: 36.169139622s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 9 out of 60 is failed (threshold=3) [..................................................XXXXXX.XXX] duration: 1m0.002396393s trace: https://jaeger.dev.storj.io/trace/51eaf95d18c2a564 FAIL (2026-04-02 00:06:55 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:05:55.554575910 (637h5m53.151346936s) [upload=713440]: Starting upload Apr 2 00:05:55.554663664 (637h5m53.15143469s) [upload=713440]: Done starting upload Apr 2 00:05:55.554808188 (637h5m53.151579223s) [upload=713440]: Uploading segments... Apr 2 00:05:55.554899119 (637h5m53.151670152s) [upload=713440]: Got next segment. Inline: true Apr 2 00:05:55.554906374 (637h5m53.151677398s) [upload=713440]: Waiting for error group managing segments... Apr 2 00:05:55.554972654 (637h5m53.151743686s) [upload=713440]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:05:55.554979172 (637h5m53.151750202s) [upload=713440]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Apr 2 00:05:55.554984144 (637h5m53.151755174s) [upload=713440]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:06:50.731115863 (637h6m48.327886896s) [upload=713440]: Done uploading segments... Apr 2 00:06:50.731163927 (637h6m48.32793495s) [upload=713440]: Upload finished. err: <nil> Duration:* 1m0.002414503s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [...............................................XXXXXXXXXXXXX] duration: 5.000867004s trace: https://jaeger.dev.storj.io/trace/4a8502250e31e208 FAIL (2026-04-02 00:06:16 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001003591s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 419.568707ms trace: https://jaeger.dev.storj.io/trace/3c28cdfe17d1a942 FAIL (2026-04-02 00:06:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:02:37.851070386 (637h2m35.44784141s) [upload=713386]: Starting upload Apr 2 00:02:37.851166355 (637h2m35.447937379s) [upload=713386]: Done starting upload Apr 2 00:02:37.851253090 (637h2m35.44802412s) [upload=713386]: Uploading segments... Apr 2 00:02:37.851793181 (637h2m35.448564196s) [upload=713386]: Got next segment. Inline: false Apr 2 00:02:37.851882102 (637h2m35.448653125s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:02:37.851889461 (637h2m35.448660484s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:02:37.880067208 (637h2m35.476838238s) [seg_pos={0 0}, upload=713386]: Begin upload segment... Apr 2 00:02:37.880168602 (637h2m35.476939634s) [seg_pos={0 0}, upload=713386]: Done begin upload segment. Apr 2 00:02:37.880207987 (637h2m35.476979021s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880214330 (637h2m35.476985339s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880292505 (637h2m35.477063531s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881027652 (637h2m35.477798688s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881378064 (637h2m35.478149089s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882097992 (637h2m35.478869026s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882586634 (637h2m35.479357658s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.883643537 (637h2m35.48041457s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884145002 (637h2m35.480916034s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884766674 (637h2m35.481537743s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885263298 (637h2m35.482034334s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885763219 (637h2m35.482534252s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886298227 (637h2m35.483069261s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886892344 (637h2m35.483663375s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.887518235 (637h2m35.484289271s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.888216955 (637h2m35.484987987s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.976004543 (637h2m35.572775567s) [upload=713386]: Next returned nil segment Apr 2 00:02:37.976025238 (637h2m35.572796273s) [upload=713386]: Waiting for error group managing segments... Apr 2 00:02:37.982357614 (637h2m35.579128639s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.982449885 (637h2m35.5792209s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983162285 (637h2m35.579933315s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983404595 (637h2m35.580175625s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983964188 (637h2m35.580735217s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.984025323 (637h2m35.58079634s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.985346526 (637h2m35.582117555s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986081391 (637h2m35.582852422s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986551623 (637h2m35.583322653s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.987396144 (637h2m35.584167177s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.997416565 (637h2m35.594187596s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005002081 (637h2m35.601773128s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005136838 (637h2m35.601907854s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005157484 (637h2m35.601928499s) [seg_pos={0 0}, upload=713386]: Satisfied cohort requirements after 13 pieces Apr 2 00:02:38.005386492 (637h2m35.602157525s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005396281 (637h2m35.60216731s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005531231 (637h2m35.602302263s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005570651 (637h2m35.602341684s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005692951 (637h2m35.602463983s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005700918 (637h2m35.602471942s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005722537 (637h2m35.60249356s) [seg_pos={0 0}, upload=713386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 2 00:02:38.005805596 (637h2m35.60257662s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:02:38.005812420 (637h2m35.602583444s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:03:07.848626111 (637h3m5.445397142s) [upload=713386]: Done uploading segments... Apr 2 00:03:07.848811089 (637h3m5.445582113s) [upload=713386]: Upload finished. err: metaclient: context deadline exceeded Duration: 419.599122ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................................................XX.XXX.] duration: 35.99713497s trace: https://jaeger.dev.storj.io/trace/4b65c7f00c3a8dfe FAIL (2026-04-02 00:06:09 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:04:22.752055593 (637h4m20.348826621s) [upload=713415]: Starting upload Apr 2 00:04:22.752142293 (637h4m20.348913314s) [upload=713415]: Done starting upload Apr 2 00:04:22.752373740 (637h4m20.349144754s) [upload=713415]: Uploading segments... Apr 2 00:04:22.752920200 (637h4m20.349691209s) [upload=713415]: Got next segment. Inline: false Apr 2 00:04:22.753003102 (637h4m20.34977413s) [upload=713415]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:04:22.753010562 (637h4m20.34978159s) [upload=713415]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:04:22.783143384 (637h4m20.379914414s) [seg_pos={0 0}, upload=713415]: Begin upload segment... Apr 2 00:04:22.783237995 (637h4m20.380009023s) [seg_pos={0 0}, upload=713415]: Done begin upload segment. Apr 2 00:04:22.783303903 (637h4m20.380074935s) [noise=true , address=145.79.227.36:4544 , piece_id=E3OMMGC5CKYJUOX5B35AYKGNSDGE4VCGFPOEUOTYPC2XRAVRXBXQ, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.783526119 (637h4m20.380297197s) [noise=true , address=45.91.116.9:4279 , piece_id=DI5LFK62FQAZ3V53XMXBV5BF5GQOXXQV2ERCZMLG6W4SFLWZJNLA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.784039849 (637h4m20.380810882s) [noise=true , address=77.90.31.18:4829 , piece_id=JSLFRG37EPS5CPLV2PZWGZNGMVD2M6FHKIH3UOEWI5NVQYVZG3OA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.784935349 (637h4m20.381706381s) [noise=true , address=45.91.116.11:4146 , piece_id=YIQM5Q5D5ATOMRCLKUS65R64V6AP2GRI2LDYGAJC7PUEL5X5R2VQ, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.785481593 (637h4m20.382252626s) [noise=true , address=145.79.227.50:4393 , piece_id=MWJG53BIMK7ZJVIDWCCFLERHNJXLDT362XVF5G5YDZRQWZYP7DVA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.786055921 (637h4m20.382826954s) [noise=true , address=145.79.227.35:4510 , piece_id=5SG3QJUTQK4DTCOM7V4WITK26PGOJDTYC73VH26DDGO66JCHUZRA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.786577360 (637h4m20.383348392s) [noise=true , address=77.90.31.21:4886 , piece_id=VO76QGDVAOKYXJSQRNG3ZDKCRSTGTJHNFAPFMJGBKVWAGGQYEMXQ, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.787104779 (637h4m20.383875806s) [noise=true , address=193.149.164.67:4023 , piece_id=R3CUHCRBGEDESHIPF4V3IDJTRIEYCKINLETW7QX2VGEGYDZSWUOQ, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.787642700 (637h4m20.384413724s) [noise=true , address=77.90.31.16:4752 , piece_id=3UUPXGPJFEOPJTUST426KIIRVNZS3EZNDNR6I62WU7EK2TTTGYDA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.788131879 (637h4m20.384902913s) [noise=true , address=45.91.116.14:4212 , piece_id=HOWBZJIMTRVXD7ZIBNHQYG4MVF24CQFZ6U5DNGG5QCAPESYE52GQ, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.788712943 (637h4m20.385483978s) [noise=true , address=193.149.164.69:4077 , piece_id=67SD52WWMRLAPWQUXDFYBH6NUN2GUIELOYDQAP276TIGJXI3NHXA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.790495292 (637h4m20.387266326s) [noise=true , address=193.149.164.70:4136 , piece_id=FNOIOGNFNTDV73LM3LPRUXBDGT344E5X6ILPDLE4NIE4EUILPT3A, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.790801985 (637h4m20.387573018s) [noise=true , address=64.49.13.137:4494 , piece_id=Y3ZLPGV6HDTQ6PTNOZL4XMDWA477P7IFILBILOUVRAIVNT2B4BAA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.793254870 (637h4m20.390025935s) [noise=true , address=167.17.72.70:4762 , piece_id=YNFEXS5LSOADFL7N5YRNS5SSN7O3CULTAJ7GV6L3AQ6QPBC7FHQA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.795129210 (637h4m20.391900248s) [noise=true , address=45.91.116.8:4059 , piece_id=RNXUSRZSYEH3U5FI67FXNMWV3UGRWWJRMMCRUFK4L4S3UCG7I65A, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.796559075 (637h4m20.393330105s) [noise=true , address=45.91.116.13:4175 , piece_id=5LJ7QONPGVNOV5RJBVREG6M7WJQ3OLAHP72LSAXBFIM3YJZ65DNQ, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.804221994 (637h4m20.400993226s) [noise=true , address=193.149.164.68:4038 , piece_id=WQQKZ4FCTWQOWBZMKU5THZFZE3FAGMSPR7LVMA6RCTHQXLLLGFHA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.808771177 (637h4m20.405542202s) [noise=true , address=145.79.227.5:4238 , piece_id=WUJT6YJXY6Z6GUJEV5LJ6IBNFZCPOHNDKTWXTN4B5KJYSYBU2XEA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.859136944 (637h4m20.455907956s) [upload=713415]: Next returned nil segment Apr 2 00:04:22.859154635 (637h4m20.45592567s) [upload=713415]: Waiting for error group managing segments... Apr 2 00:04:22.864190905 (637h4m20.460961929s) [noise=true , address=45.91.116.8:4059 , piece_id=RNXUSRZSYEH3U5FI67FXNMWV3UGRWWJRMMCRUFK4L4S3UCG7I65A, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.865233224 (637h4m20.462004248s) [noise=true , address=145.79.227.5:4238 , piece_id=WUJT6YJXY6Z6GUJEV5LJ6IBNFZCPOHNDKTWXTN4B5KJYSYBU2XEA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.865328660 (637h4m20.462099686s) [noise=true , address=45.91.116.9:4279 , piece_id=DI5LFK62FQAZ3V53XMXBV5BF5GQOXXQV2ERCZMLG6W4SFLWZJNLA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.865956793 (637h4m20.462727822s) [noise=true , address=77.90.31.16:4752 , piece_id=3UUPXGPJFEOPJTUST426KIIRVNZS3EZNDNR6I62WU7EK2TTTGYDA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.868160098 (637h4m20.464931123s) [noise=true , address=145.79.227.35:4510 , piece_id=5SG3QJUTQK4DTCOM7V4WITK26PGOJDTYC73VH26DDGO66JCHUZRA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.868583222 (637h4m20.465354253s) [noise=true , address=45.91.116.13:4175 , piece_id=5LJ7QONPGVNOV5RJBVREG6M7WJQ3OLAHP72LSAXBFIM3YJZ65DNQ, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.877818662 (637h4m20.47458969s) [noise=true , address=145.79.227.36:4544 , piece_id=E3OMMGC5CKYJUOX5B35AYKGNSDGE4VCGFPOEUOTYPC2XRAVRXBXQ, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.878065899 (637h4m20.474836927s) [noise=true , address=77.90.31.18:4829 , piece_id=JSLFRG37EPS5CPLV2PZWGZNGMVD2M6FHKIH3UOEWI5NVQYVZG3OA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.878562340 (637h4m20.475333372s) [noise=true , address=45.91.116.14:4212 , piece_id=HOWBZJIMTRVXD7ZIBNHQYG4MVF24CQFZ6U5DNGG5QCAPESYE52GQ, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.879067238 (637h4m20.475838265s) [noise=true , address=77.90.31.21:4886 , piece_id=VO76QGDVAOKYXJSQRNG3ZDKCRSTGTJHNFAPFMJGBKVWAGGQYEMXQ, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.880940501 (637h4m20.477711531s) [noise=true , address=145.79.227.50:4393 , piece_id=MWJG53BIMK7ZJVIDWCCFLERHNJXLDT362XVF5G5YDZRQWZYP7DVA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.881924153 (637h4m20.478695178s) [noise=true , address=45.91.116.11:4146 , piece_id=YIQM5Q5D5ATOMRCLKUS65R64V6AP2GRI2LDYGAJC7PUEL5X5R2VQ, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:23.182938845 (637h4m20.779709859s) [noise=true , address=167.17.72.70:4762 , piece_id=YNFEXS5LSOADFL7N5YRNS5SSN7O3CULTAJ7GV6L3AQ6QPBC7FHQA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:23.389241087 (637h4m20.98601211s) [noise=true , address=193.149.164.67:4023 , piece_id=R3CUHCRBGEDESHIPF4V3IDJTRIEYCKINLETW7QX2VGEGYDZSWUOQ, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:23.389433377 (637h4m20.986204399s) [seg_pos={0 0}, upload=713415]: Satisfied cohort requirements after 14 pieces Apr 2 00:04:23.389804584 (637h4m20.98657561s) [noise=true , address=193.149.164.68:4038 , piece_id=WQQKZ4FCTWQOWBZMKU5THZFZE3FAGMSPR7LVMA6RCTHQXLLLGFHA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LvCkhgeYiXJLJtJSXmbrmWXxMCUTePAT6Ww6zaCbP2JyUhK6k): protocol: expected piece hash; context canceled; EOF Apr 2 00:04:23.389825318 (637h4m20.986596345s) [noise=true , address=193.149.164.68:4038 , piece_id=WQQKZ4FCTWQOWBZMKU5THZFZE3FAGMSPR7LVMA6RCTHQXLLLGFHA, seg_pos={0 0}, upload=713415]: Long tail cancellation due to optimal threshold Apr 2 00:04:23.390075793 (637h4m20.986846826s) [noise=true , address=64.49.13.137:4494 , piece_id=Y3ZLPGV6HDTQ6PTNOZL4XMDWA477P7IFILBILOUVRAIVNT2B4BAA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TX66fDVv3qu4yBGx9FNZTiArNeUnqxjMYwb3VGqYCQRe8RwoU): protocol: expected piece hash; context canceled; EOF Apr 2 00:04:23.390082975 (637h4m20.986853998s) [noise=true , address=64.49.13.137:4494 , piece_id=Y3ZLPGV6HDTQ6PTNOZL4XMDWA477P7IFILBILOUVRAIVNT2B4BAA, seg_pos={0 0}, upload=713415]: Long tail cancellation due to optimal threshold Apr 2 00:04:23.390452875 (637h4m20.98722391s) [noise=true , address=193.149.164.69:4077 , piece_id=67SD52WWMRLAPWQUXDFYBH6NUN2GUIELOYDQAP276TIGJXI3NHXA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qcwvrgkkT6Q8pVEuUu7fA4wwHXNidvCNnKhrU3DpdqAHKGtLG): protocol: expected piece hash; context canceled; EOF Apr 2 00:04:23.390466752 (637h4m20.987237821s) [noise=true , address=193.149.164.69:4077 , piece_id=67SD52WWMRLAPWQUXDFYBH6NUN2GUIELOYDQAP276TIGJXI3NHXA, seg_pos={0 0}, upload=713415]: Long tail cancellation due to optimal threshold Apr 2 00:04:23.390792198 (637h4m20.987563321s) [noise=true , address=193.149.164.70:4136 , piece_id=FNOIOGNFNTDV73LM3LPRUXBDGT344E5X6ILPDLE4NIE4EUILPT3A, seg_pos={0 0}, upload=713415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YVyrEWNRczm6PHNwrAp58CdCuZPxSSHpzAqZ3F914MqvUNjuw): protocol: expected piece hash; context canceled; EOF Apr 2 00:04:23.390806838 (637h4m20.987577907s) [noise=true , address=193.149.164.70:4136 , piece_id=FNOIOGNFNTDV73LM3LPRUXBDGT344E5X6ILPDLE4NIE4EUILPT3A, seg_pos={0 0}, upload=713415]: Long tail cancellation due to optimal threshold Apr 2 00:04:23.390851162 (637h4m20.987622206s) [seg_pos={0 0}, upload=713415]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 2 00:04:23.390967742 (637h4m20.987738776s) [upload=713415]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:04:23.390974630 (637h4m20.987745661s) [upload=713415]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:05:33.093755528 (637h5m30.690526555s) [upload=713415]: Done uploading segments... Apr 2 00:05:33.093876891 (637h5m30.69064792s) [upload=713415]: Upload finished. err: metaclient: context deadline exceeded Duration: 35.997155883s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 8 out of 60 is failed (threshold=3) [...................................................XXXXXX.XX] duration: 1m0.002717182s trace: https://jaeger.dev.storj.io/trace/604b86a0b34306c9 FAIL (2026-04-02 00:05:55 UTC) operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:04:55.483311904 (637h4m53.08008294s) [upload=713422]: Starting upload Apr 2 00:04:55.483413279 (637h4m53.080184313s) [upload=713422]: Done starting upload Apr 2 00:04:55.483511056 (637h4m53.080282066s) [upload=713422]: Uploading segments... Apr 2 00:04:55.483586636 (637h4m53.080357667s) [upload=713422]: Got next segment. Inline: true Apr 2 00:04:55.483592806 (637h4m53.080363836s) [upload=713422]: Waiting for error group managing segments... Apr 2 00:04:55.483650769 (637h4m53.080421797s) [upload=713422]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:04:55.483656918 (637h4m53.080427946s) [upload=713422]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Apr 2 00:04:55.483660573 (637h4m53.080431601s) [upload=713422]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:04:55.545898532 (637h4m53.142669547s) [upload=713422]: Done uploading segments... Apr 2 00:04:55.545923311 (637h4m53.142694329s) [upload=713422]: Upload finished. err: <nil> Duration:* 1m0.002751341s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [................................................XXXXXXXXXXXX] duration: 5.001073111s trace: https://jaeger.dev.storj.io/trace/27d100b35301020b FAIL (2026-04-02 00:05:40 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001098098s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................................................XX.XXX] duration: 1m10.378040264s trace: https://jaeger.dev.storj.io/trace/4b65c7f00c3a8dfe FAIL (2026-04-02 00:05:33 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:04:22.752055593 (637h4m20.348826621s) [upload=713415]: Starting upload Apr 2 00:04:22.752142293 (637h4m20.348913314s) [upload=713415]: Done starting upload Apr 2 00:04:22.752373740 (637h4m20.349144754s) [upload=713415]: Uploading segments... Apr 2 00:04:22.752920200 (637h4m20.349691209s) [upload=713415]: Got next segment. Inline: false Apr 2 00:04:22.753003102 (637h4m20.34977413s) [upload=713415]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:04:22.753010562 (637h4m20.34978159s) [upload=713415]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:04:22.783143384 (637h4m20.379914414s) [seg_pos={0 0}, upload=713415]: Begin upload segment... Apr 2 00:04:22.783237995 (637h4m20.380009023s) [seg_pos={0 0}, upload=713415]: Done begin upload segment. Apr 2 00:04:22.783303903 (637h4m20.380074935s) [noise=true , address=145.79.227.36:4544 , piece_id=E3OMMGC5CKYJUOX5B35AYKGNSDGE4VCGFPOEUOTYPC2XRAVRXBXQ, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.783526119 (637h4m20.380297197s) [noise=true , address=45.91.116.9:4279 , piece_id=DI5LFK62FQAZ3V53XMXBV5BF5GQOXXQV2ERCZMLG6W4SFLWZJNLA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.784039849 (637h4m20.380810882s) [noise=true , address=77.90.31.18:4829 , piece_id=JSLFRG37EPS5CPLV2PZWGZNGMVD2M6FHKIH3UOEWI5NVQYVZG3OA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.784935349 (637h4m20.381706381s) [noise=true , address=45.91.116.11:4146 , piece_id=YIQM5Q5D5ATOMRCLKUS65R64V6AP2GRI2LDYGAJC7PUEL5X5R2VQ, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.785481593 (637h4m20.382252626s) [noise=true , address=145.79.227.50:4393 , piece_id=MWJG53BIMK7ZJVIDWCCFLERHNJXLDT362XVF5G5YDZRQWZYP7DVA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.786055921 (637h4m20.382826954s) [noise=true , address=145.79.227.35:4510 , piece_id=5SG3QJUTQK4DTCOM7V4WITK26PGOJDTYC73VH26DDGO66JCHUZRA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.786577360 (637h4m20.383348392s) [noise=true , address=77.90.31.21:4886 , piece_id=VO76QGDVAOKYXJSQRNG3ZDKCRSTGTJHNFAPFMJGBKVWAGGQYEMXQ, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.787104779 (637h4m20.383875806s) [noise=true , address=193.149.164.67:4023 , piece_id=R3CUHCRBGEDESHIPF4V3IDJTRIEYCKINLETW7QX2VGEGYDZSWUOQ, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.787642700 (637h4m20.384413724s) [noise=true , address=77.90.31.16:4752 , piece_id=3UUPXGPJFEOPJTUST426KIIRVNZS3EZNDNR6I62WU7EK2TTTGYDA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.788131879 (637h4m20.384902913s) [noise=true , address=45.91.116.14:4212 , piece_id=HOWBZJIMTRVXD7ZIBNHQYG4MVF24CQFZ6U5DNGG5QCAPESYE52GQ, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.788712943 (637h4m20.385483978s) [noise=true , address=193.149.164.69:4077 , piece_id=67SD52WWMRLAPWQUXDFYBH6NUN2GUIELOYDQAP276TIGJXI3NHXA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.790495292 (637h4m20.387266326s) [noise=true , address=193.149.164.70:4136 , piece_id=FNOIOGNFNTDV73LM3LPRUXBDGT344E5X6ILPDLE4NIE4EUILPT3A, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.790801985 (637h4m20.387573018s) [noise=true , address=64.49.13.137:4494 , piece_id=Y3ZLPGV6HDTQ6PTNOZL4XMDWA477P7IFILBILOUVRAIVNT2B4BAA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.793254870 (637h4m20.390025935s) [noise=true , address=167.17.72.70:4762 , piece_id=YNFEXS5LSOADFL7N5YRNS5SSN7O3CULTAJ7GV6L3AQ6QPBC7FHQA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.795129210 (637h4m20.391900248s) [noise=true , address=45.91.116.8:4059 , piece_id=RNXUSRZSYEH3U5FI67FXNMWV3UGRWWJRMMCRUFK4L4S3UCG7I65A, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.796559075 (637h4m20.393330105s) [noise=true , address=45.91.116.13:4175 , piece_id=5LJ7QONPGVNOV5RJBVREG6M7WJQ3OLAHP72LSAXBFIM3YJZ65DNQ, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.804221994 (637h4m20.400993226s) [noise=true , address=193.149.164.68:4038 , piece_id=WQQKZ4FCTWQOWBZMKU5THZFZE3FAGMSPR7LVMA6RCTHQXLLLGFHA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.808771177 (637h4m20.405542202s) [noise=true , address=145.79.227.5:4238 , piece_id=WUJT6YJXY6Z6GUJEV5LJ6IBNFZCPOHNDKTWXTN4B5KJYSYBU2XEA, seg_pos={0 0}, upload=713415]: Uploading piece... Apr 2 00:04:22.859136944 (637h4m20.455907956s) [upload=713415]: Next returned nil segment Apr 2 00:04:22.859154635 (637h4m20.45592567s) [upload=713415]: Waiting for error group managing segments... Apr 2 00:04:22.864190905 (637h4m20.460961929s) [noise=true , address=45.91.116.8:4059 , piece_id=RNXUSRZSYEH3U5FI67FXNMWV3UGRWWJRMMCRUFK4L4S3UCG7I65A, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.865233224 (637h4m20.462004248s) [noise=true , address=145.79.227.5:4238 , piece_id=WUJT6YJXY6Z6GUJEV5LJ6IBNFZCPOHNDKTWXTN4B5KJYSYBU2XEA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.865328660 (637h4m20.462099686s) [noise=true , address=45.91.116.9:4279 , piece_id=DI5LFK62FQAZ3V53XMXBV5BF5GQOXXQV2ERCZMLG6W4SFLWZJNLA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.865956793 (637h4m20.462727822s) [noise=true , address=77.90.31.16:4752 , piece_id=3UUPXGPJFEOPJTUST426KIIRVNZS3EZNDNR6I62WU7EK2TTTGYDA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.868160098 (637h4m20.464931123s) [noise=true , address=145.79.227.35:4510 , piece_id=5SG3QJUTQK4DTCOM7V4WITK26PGOJDTYC73VH26DDGO66JCHUZRA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.868583222 (637h4m20.465354253s) [noise=true , address=45.91.116.13:4175 , piece_id=5LJ7QONPGVNOV5RJBVREG6M7WJQ3OLAHP72LSAXBFIM3YJZ65DNQ, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.877818662 (637h4m20.47458969s) [noise=true , address=145.79.227.36:4544 , piece_id=E3OMMGC5CKYJUOX5B35AYKGNSDGE4VCGFPOEUOTYPC2XRAVRXBXQ, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.878065899 (637h4m20.474836927s) [noise=true , address=77.90.31.18:4829 , piece_id=JSLFRG37EPS5CPLV2PZWGZNGMVD2M6FHKIH3UOEWI5NVQYVZG3OA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.878562340 (637h4m20.475333372s) [noise=true , address=45.91.116.14:4212 , piece_id=HOWBZJIMTRVXD7ZIBNHQYG4MVF24CQFZ6U5DNGG5QCAPESYE52GQ, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.879067238 (637h4m20.475838265s) [noise=true , address=77.90.31.21:4886 , piece_id=VO76QGDVAOKYXJSQRNG3ZDKCRSTGTJHNFAPFMJGBKVWAGGQYEMXQ, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.880940501 (637h4m20.477711531s) [noise=true , address=145.79.227.50:4393 , piece_id=MWJG53BIMK7ZJVIDWCCFLERHNJXLDT362XVF5G5YDZRQWZYP7DVA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:22.881924153 (637h4m20.478695178s) [noise=true , address=45.91.116.11:4146 , piece_id=YIQM5Q5D5ATOMRCLKUS65R64V6AP2GRI2LDYGAJC7PUEL5X5R2VQ, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:23.182938845 (637h4m20.779709859s) [noise=true , address=167.17.72.70:4762 , piece_id=YNFEXS5LSOADFL7N5YRNS5SSN7O3CULTAJ7GV6L3AQ6QPBC7FHQA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:23.389241087 (637h4m20.98601211s) [noise=true , address=193.149.164.67:4023 , piece_id=R3CUHCRBGEDESHIPF4V3IDJTRIEYCKINLETW7QX2VGEGYDZSWUOQ, seg_pos={0 0}, upload=713415]: Done uploading piece. err: <nil> Apr 2 00:04:23.389433377 (637h4m20.986204399s) [seg_pos={0 0}, upload=713415]: Satisfied cohort requirements after 14 pieces Apr 2 00:04:23.389804584 (637h4m20.98657561s) [noise=true , address=193.149.164.68:4038 , piece_id=WQQKZ4FCTWQOWBZMKU5THZFZE3FAGMSPR7LVMA6RCTHQXLLLGFHA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LvCkhgeYiXJLJtJSXmbrmWXxMCUTePAT6Ww6zaCbP2JyUhK6k): protocol: expected piece hash; context canceled; EOF Apr 2 00:04:23.389825318 (637h4m20.986596345s) [noise=true , address=193.149.164.68:4038 , piece_id=WQQKZ4FCTWQOWBZMKU5THZFZE3FAGMSPR7LVMA6RCTHQXLLLGFHA, seg_pos={0 0}, upload=713415]: Long tail cancellation due to optimal threshold Apr 2 00:04:23.390075793 (637h4m20.986846826s) [noise=true , address=64.49.13.137:4494 , piece_id=Y3ZLPGV6HDTQ6PTNOZL4XMDWA477P7IFILBILOUVRAIVNT2B4BAA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TX66fDVv3qu4yBGx9FNZTiArNeUnqxjMYwb3VGqYCQRe8RwoU): protocol: expected piece hash; context canceled; EOF Apr 2 00:04:23.390082975 (637h4m20.986853998s) [noise=true , address=64.49.13.137:4494 , piece_id=Y3ZLPGV6HDTQ6PTNOZL4XMDWA477P7IFILBILOUVRAIVNT2B4BAA, seg_pos={0 0}, upload=713415]: Long tail cancellation due to optimal threshold Apr 2 00:04:23.390452875 (637h4m20.98722391s) [noise=true , address=193.149.164.69:4077 , piece_id=67SD52WWMRLAPWQUXDFYBH6NUN2GUIELOYDQAP276TIGJXI3NHXA, seg_pos={0 0}, upload=713415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qcwvrgkkT6Q8pVEuUu7fA4wwHXNidvCNnKhrU3DpdqAHKGtLG): protocol: expected piece hash; context canceled; EOF Apr 2 00:04:23.390466752 (637h4m20.987237821s) [noise=true , address=193.149.164.69:4077 , piece_id=67SD52WWMRLAPWQUXDFYBH6NUN2GUIELOYDQAP276TIGJXI3NHXA, seg_pos={0 0}, upload=713415]: Long tail cancellation due to optimal threshold Apr 2 00:04:23.390792198 (637h4m20.987563321s) [noise=true , address=193.149.164.70:4136 , piece_id=FNOIOGNFNTDV73LM3LPRUXBDGT344E5X6ILPDLE4NIE4EUILPT3A, seg_pos={0 0}, upload=713415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YVyrEWNRczm6PHNwrAp58CdCuZPxSSHpzAqZ3F914MqvUNjuw): protocol: expected piece hash; context canceled; EOF Apr 2 00:04:23.390806838 (637h4m20.987577907s) [noise=true , address=193.149.164.70:4136 , piece_id=FNOIOGNFNTDV73LM3LPRUXBDGT344E5X6ILPDLE4NIE4EUILPT3A, seg_pos={0 0}, upload=713415]: Long tail cancellation due to optimal threshold Apr 2 00:04:23.390851162 (637h4m20.987622206s) [seg_pos={0 0}, upload=713415]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 2 00:04:23.390967742 (637h4m20.987738776s) [upload=713415]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:04:23.390974630 (637h4m20.987745661s) [upload=713415]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:05:33.093755528 (637h5m30.690526555s) [upload=713415]: Done uploading segments... Apr 2 00:05:33.093876891 (637h5m30.69064792s) [upload=713415]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m10.378055221s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 562.340192ms trace: https://jaeger.dev.storj.io/trace/3c28cdfe17d1a942 FAIL (2026-04-02 00:04:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:02:37.851070386 (637h2m35.44784141s) [upload=713386]: Starting upload Apr 2 00:02:37.851166355 (637h2m35.447937379s) [upload=713386]: Done starting upload Apr 2 00:02:37.851253090 (637h2m35.44802412s) [upload=713386]: Uploading segments... Apr 2 00:02:37.851793181 (637h2m35.448564196s) [upload=713386]: Got next segment. Inline: false Apr 2 00:02:37.851882102 (637h2m35.448653125s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:02:37.851889461 (637h2m35.448660484s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:02:37.880067208 (637h2m35.476838238s) [seg_pos={0 0}, upload=713386]: Begin upload segment... Apr 2 00:02:37.880168602 (637h2m35.476939634s) [seg_pos={0 0}, upload=713386]: Done begin upload segment. Apr 2 00:02:37.880207987 (637h2m35.476979021s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880214330 (637h2m35.476985339s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880292505 (637h2m35.477063531s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881027652 (637h2m35.477798688s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881378064 (637h2m35.478149089s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882097992 (637h2m35.478869026s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882586634 (637h2m35.479357658s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.883643537 (637h2m35.48041457s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884145002 (637h2m35.480916034s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884766674 (637h2m35.481537743s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885263298 (637h2m35.482034334s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885763219 (637h2m35.482534252s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886298227 (637h2m35.483069261s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886892344 (637h2m35.483663375s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.887518235 (637h2m35.484289271s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.888216955 (637h2m35.484987987s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.976004543 (637h2m35.572775567s) [upload=713386]: Next returned nil segment Apr 2 00:02:37.976025238 (637h2m35.572796273s) [upload=713386]: Waiting for error group managing segments... Apr 2 00:02:37.982357614 (637h2m35.579128639s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.982449885 (637h2m35.5792209s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983162285 (637h2m35.579933315s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983404595 (637h2m35.580175625s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983964188 (637h2m35.580735217s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.984025323 (637h2m35.58079634s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.985346526 (637h2m35.582117555s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986081391 (637h2m35.582852422s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986551623 (637h2m35.583322653s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.987396144 (637h2m35.584167177s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.997416565 (637h2m35.594187596s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005002081 (637h2m35.601773128s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005136838 (637h2m35.601907854s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005157484 (637h2m35.601928499s) [seg_pos={0 0}, upload=713386]: Satisfied cohort requirements after 13 pieces Apr 2 00:02:38.005386492 (637h2m35.602157525s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005396281 (637h2m35.60216731s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005531231 (637h2m35.602302263s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005570651 (637h2m35.602341684s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005692951 (637h2m35.602463983s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005700918 (637h2m35.602471942s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005722537 (637h2m35.60249356s) [seg_pos={0 0}, upload=713386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 2 00:02:38.005805596 (637h2m35.60257662s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:02:38.005812420 (637h2m35.602583444s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:03:07.848626111 (637h3m5.445397142s) [upload=713386]: Done uploading segments... Apr 2 00:03:07.848811089 (637h3m5.445582113s) [upload=713386]: Upload finished. err: metaclient: context deadline exceeded Duration: 562.373284ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 7 out of 60 is failed (threshold=3) [....................................................XXXXXX.X] duration: 1m0.000973668s trace: https://jaeger.dev.storj.io/trace/2f88c9af2d84979a FAIL (2026-04-02 00:04:55 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:03:55.405257680 (637h3m53.002028704s) [upload=713407]: Starting upload Apr 2 00:03:55.405343437 (637h3m53.002114471s) [upload=713407]: Done starting upload Apr 2 00:03:55.405516884 (637h3m53.002287893s) [upload=713407]: Uploading segments... Apr 2 00:03:55.405640530 (637h3m53.002411567s) [upload=713407]: Got next segment. Inline: true Apr 2 00:03:55.405650304 (637h3m53.002421334s) [upload=713407]: Waiting for error group managing segments... Apr 2 00:03:55.405744671 (637h3m53.002515706s) [upload=713407]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:03:55.405752503 (637h3m53.002523528s) [upload=713407]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Apr 2 00:03:55.405756940 (637h3m53.002527964s) [upload=713407]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:04:55.362078363 (637h4m52.958849394s) [upload=713407]: Done uploading segments... Apr 2 00:04:55.362155792 (637h4m52.958926807s) [upload=713407]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1m0.000989112s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.................................................XXXXXXXXXXX] duration: 5.001353251s trace: https://jaeger.dev.storj.io/trace/34e8bd7acd19b294 FAIL (2026-04-02 00:04:40 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001418318s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................................XX.XX] duration: 2m56.59236469s trace: https://jaeger.dev.storj.io/trace/5656d4837ff8521f FAIL (2026-04-02 00:04:22 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:01:26.092500845 (637h1m23.689271868s) [upload=713365]: Starting upload Apr 2 00:01:26.092659356 (637h1m23.68943038s) [upload=713365]: Done starting upload Apr 2 00:01:26.092745729 (637h1m23.689516753s) [upload=713365]: Uploading segments... Apr 2 00:01:26.093199291 (637h1m23.689970341s) [upload=713365]: Got next segment. Inline: false Apr 2 00:01:26.093325007 (637h1m23.690096041s) [upload=713365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:01:26.093333472 (637h1m23.690104506s) [upload=713365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:01:26.138469453 (637h1m23.735240476s) [seg_pos={0 0}, upload=713365]: Begin upload segment... Apr 2 00:01:26.138534594 (637h1m23.735305621s) [seg_pos={0 0}, upload=713365]: Done begin upload segment. Apr 2 00:01:26.138565698 (637h1m23.735336721s) [noise=true , address=193.149.164.66:4993 , piece_id=BZDCB4CN5ECUOJQBQKA73VTC6ZTT6DTNGFQYEP7ZVMNVPRCFBP2A, seg_pos={0 0}, upload=713365]: Uploading piece... Apr 2 00:01:26.139223140 (637h1m23.735994178s) [noise=true , address=193.149.164.69:4094 , piece_id=B4UPJT2ESNJNGTV5FSFAQU7G6UGWJ5L3MCGDOCLKWQHQOZCFDIKA, seg_pos={0 0}, upload=713365]: Uploading piece... Apr 2 00:01:26.139485348 (637h1m23.736256371s) [noise=true , address=167.17.72.69:4730 , piece_id=P6BPGS5PQRYLFKHJTK3TPYIAL7ZIUO6SSGU7SAXIBHFSIXQYQJAA, seg_pos={0 0}, upload=713365]: Uploading piece... Apr 2 00:01:26.140036236 (637h1m23.736807262s) [noise=true , address=45.91.116.11:4125 , piece_id=6BA7ILIGIF4GUATONRX7GBE5C5CP2OQIMHTKKAO4TDQF7EUAMFUA, seg_pos={0 0}, upload=713365]: Uploading piece... Apr 2 00:01:26.140146849 (637h1m23.736917882s) [noise=true , address=45.91.116.7:4027 , piece_id=5PVVDPE5K7TBK4LOLLVBFZ6GFS7YUZHNS3RPWDLP6E7UK7QAP4UA, seg_pos={0 0}, upload=713365]: Uploading piece... Apr 2 00:01:26.140618459 (637h1m23.737389493s) [noise=true , address=167.17.72.68:4646 , piece_id=HCL4QMIPXCPASOFB65C4GJXANWCAFZN7V6VGL4CA7TU3COCODZYQ, seg_pos={0 0}, upload=713365]: Uploading piece... Apr 2 00:01:26.140729192 (637h1m23.737500228s) [noise=true , address=167.17.72.70:4775 , piece_id=JMC3EUY6TWNYFITU2WS43HWUYMZPKU4W7JVJ6Z7AVTTCAKB3TH2A, seg_pos={0 0}, upload=713365]: Uploading piece... Apr 2 00:01:26.141222840 (637h1m23.737993873s) [noise=true , address=167.17.72.71:4796 , piece_id=ODQIDCVJV34G57YKN7EGNN4HJA4EPYU4CCFNMTKX662BG5GNRENA, seg_pos={0 0}, upload=713365]: Uploading piece... Apr 2 00:01:26.141231227 (637h1m23.73800225s) [noise=true , address=145.79.227.30:4785 , piece_id=RABH3BOSWR36RUWDBVZYN37A4CLSDGJ6XN3SFMFYIMEIVYTC7SNQ, seg_pos={0 0}, upload=713365]: Uploading piece... Apr 2 00:01:26.141724620 (637h1m23.738495653s) [noise=true , address=145.79.227.6:4279 , piece_id=7O5OMGG47N2WSSUMFBPESUTHE7FFPGAZO5FLLSBA6U6ZSHIO5GKQ, seg_pos={0 0}, upload=713365]: Uploading piece... Apr 2 00:01:26.141744549 (637h1m23.738515578s) [noise=true , address=193.149.164.70:4130 , piece_id=AKA4PAWPDWLZZ6JLPTBIU75VVFCX3H3F4DFJCULMMX7T2NH4ED3Q, seg_pos={0 0}, upload=713365]: Uploading piece... Apr 2 00:01:26.142195264 (637h1m23.738966291s) [noise=true , address=193.149.164.68:4049 , piece_id=6QJKZ4DVIVBNCUQ2APCNIIL4EHXYPOHQUSVXT4LWJI4LGZAZNKCA, seg_pos={0 0}, upload=713365]: Uploading piece... Apr 2 00:01:26.142428945 (637h1m23.739199968s) [noise=true , address=145.79.227.13:4826 , piece_id=ABKYY2FG4CT3LNCRP2IKFVUVY6IJLUMCXRXHPMUVCLX2AXZNGGEQ, seg_pos={0 0}, upload=713365]: Uploading piece... Apr 2 00:01:26.142884723 (637h1m23.739655746s) [noise=true , address=145.79.227.29:4949 , piece_id=GRAC5QTBK6XCXKUKSX5H27D7KSHHJKRWLTCZZJI7CMQUYODUO6IA, seg_pos={0 0}, upload=713365]: Uploading piece... Apr 2 00:01:26.143374376 (637h1m23.740145399s) [noise=true , address=77.90.31.16:4742 , piece_id=G2QSNUA6MGO7UCQ62WOLRASLJITXMLQKYP3FB4ZA34SYVALQVBKQ, seg_pos={0 0}, upload=713365]: Uploading piece... Apr 2 00:01:26.143879622 (637h1m23.740650664s) [noise=true , address=145.79.227.26:4672 , piece_id=NFCXPEMRFNKLAYXJLNQJLPGWXRCXZFNWK7BOOIZLLDJ6ORZIXC5Q, seg_pos={0 0}, upload=713365]: Uploading piece... Apr 2 00:01:26.146243585 (637h1m23.743026912s) [noise=true , address=64.49.13.170:4724 , piece_id=ALSGCSFXM43ENEBFFRYNLDNX45LKYM6HDHVHA6GVKGMCNZXNPJDA, seg_pos={0 0}, upload=713365]: Uploading piece... Apr 2 00:01:26.147064001 (637h1m23.743835047s) [noise=true , address=145.79.227.52:4470 , piece_id=VQR2HRUGNPU7ERJPM74USEZIWWGWWP4YDHFCAAATXXRJJZ4EAVGA, seg_pos={0 0}, upload=713365]: Uploading piece... Apr 2 00:01:26.214325316 (637h1m23.811096333s) [upload=713365]: Next returned nil segment Apr 2 00:01:26.214337871 (637h1m23.811108894s) [upload=713365]: Waiting for error group managing segments... Apr 2 00:01:26.220948240 (637h1m23.817719264s) [noise=true , address=145.79.227.6:4279 , piece_id=7O5OMGG47N2WSSUMFBPESUTHE7FFPGAZO5FLLSBA6U6ZSHIO5GKQ, seg_pos={0 0}, upload=713365]: Done uploading piece. err: <nil> Apr 2 00:01:26.231427416 (637h1m23.828198442s) [noise=true , address=45.91.116.11:4125 , piece_id=6BA7ILIGIF4GUATONRX7GBE5C5CP2OQIMHTKKAO4TDQF7EUAMFUA, seg_pos={0 0}, upload=713365]: Done uploading piece. err: <nil> Apr 2 00:01:26.231477300 (637h1m23.828248323s) [noise=true , address=145.79.227.30:4785 , piece_id=RABH3BOSWR36RUWDBVZYN37A4CLSDGJ6XN3SFMFYIMEIVYTC7SNQ, seg_pos={0 0}, upload=713365]: Done uploading piece. err: <nil> Apr 2 00:01:26.234309120 (637h1m23.831080146s) [noise=true , address=145.79.227.26:4672 , piece_id=NFCXPEMRFNKLAYXJLNQJLPGWXRCXZFNWK7BOOIZLLDJ6ORZIXC5Q, seg_pos={0 0}, upload=713365]: Done uploading piece. err: <nil> Apr 2 00:01:26.234345127 (637h1m23.831116152s) [noise=true , address=45.91.116.7:4027 , piece_id=5PVVDPE5K7TBK4LOLLVBFZ6GFS7YUZHNS3RPWDLP6E7UK7QAP4UA, seg_pos={0 0}, upload=713365]: Done uploading piece. err: <nil> Apr 2 00:01:26.236917229 (637h1m23.833688242s) [noise=true , address=145.79.227.29:4949 , piece_id=GRAC5QTBK6XCXKUKSX5H27D7KSHHJKRWLTCZZJI7CMQUYODUO6IA, seg_pos={0 0}, upload=713365]: Done uploading piece. err: <nil> Apr 2 00:01:26.238884579 (637h1m23.835655609s) [noise=true , address=77.90.31.16:4742 , piece_id=G2QSNUA6MGO7UCQ62WOLRASLJITXMLQKYP3FB4ZA34SYVALQVBKQ, seg_pos={0 0}, upload=713365]: Done uploading piece. err: <nil> Apr 2 00:01:26.246574062 (637h1m23.843345076s) [noise=true , address=145.79.227.52:4470 , piece_id=VQR2HRUGNPU7ERJPM74USEZIWWGWWP4YDHFCAAATXXRJJZ4EAVGA, seg_pos={0 0}, upload=713365]: Done uploading piece. err: <nil> Apr 2 00:01:26.490574255 (637h1m24.087345279s) [noise=true , address=167.17.72.68:4646 , piece_id=HCL4QMIPXCPASOFB65C4GJXANWCAFZN7V6VGL4CA7TU3COCODZYQ, seg_pos={0 0}, upload=713365]: Done uploading piece. err: <nil> Apr 2 00:01:26.512910374 (637h1m24.109681397s) [noise=true , address=167.17.72.71:4796 , piece_id=ODQIDCVJV34G57YKN7EGNN4HJA4EPYU4CCFNMTKX662BG5GNRENA, seg_pos={0 0}, upload=713365]: Done uploading piece. err: <nil> Apr 2 00:01:26.527347874 (637h1m24.124118918s) [noise=true , address=167.17.72.69:4730 , piece_id=P6BPGS5PQRYLFKHJTK3TPYIAL7ZIUO6SSGU7SAXIBHFSIXQYQJAA, seg_pos={0 0}, upload=713365]: Done uploading piece. err: <nil> Apr 2 00:01:26.643131883 (637h1m24.239902907s) [noise=true , address=167.17.72.70:4775 , piece_id=JMC3EUY6TWNYFITU2WS43HWUYMZPKU4W7JVJ6Z7AVTTCAKB3TH2A, seg_pos={0 0}, upload=713365]: Done uploading piece. err: <nil> Apr 2 00:01:26.724695893 (637h1m24.321466923s) [noise=true , address=193.149.164.68:4049 , piece_id=6QJKZ4DVIVBNCUQ2APCNIIL4EHXYPOHQUSVXT4LWJI4LGZAZNKCA, seg_pos={0 0}, upload=713365]: Done uploading piece. err: <nil> Apr 2 00:01:26.728078186 (637h1m24.324849211s) [noise=true , address=64.49.13.170:4724 , piece_id=ALSGCSFXM43ENEBFFRYNLDNX45LKYM6HDHVHA6GVKGMCNZXNPJDA, seg_pos={0 0}, upload=713365]: Done uploading piece. err: <nil> Apr 2 00:01:26.728153301 (637h1m24.324924331s) [seg_pos={0 0}, upload=713365]: Satisfied cohort requirements after 14 pieces Apr 2 00:01:26.728473418 (637h1m24.325244454s) [noise=true , address=193.149.164.69:4094 , piece_id=B4UPJT2ESNJNGTV5FSFAQU7G6UGWJ5L3MCGDOCLKWQHQOZCFDIKA, seg_pos={0 0}, upload=713365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kdx965fb4YxXVnbx9y4WGGnmSDT7qvg1coDRe5JWqYXAdy1fP): protocol: expected piece hash; context canceled; EOF Apr 2 00:01:26.728490274 (637h1m24.325261284s) [noise=true , address=193.149.164.69:4094 , piece_id=B4UPJT2ESNJNGTV5FSFAQU7G6UGWJ5L3MCGDOCLKWQHQOZCFDIKA, seg_pos={0 0}, upload=713365]: Long tail cancellation due to optimal threshold Apr 2 00:01:26.729050224 (637h1m24.325821266s) [noise=true , address=193.149.164.70:4130 , piece_id=AKA4PAWPDWLZZ6JLPTBIU75VVFCX3H3F4DFJCULMMX7T2NH4ED3Q, seg_pos={0 0}, upload=713365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FeBbH8G9Bx9QnmHyVcHoCkAWhFPRyk7JEn1boPKsZC7Ya74xa): protocol: expected piece hash; context canceled; EOF Apr 2 00:01:26.729063764 (637h1m24.3258348s) [noise=true , address=193.149.164.70:4130 , piece_id=AKA4PAWPDWLZZ6JLPTBIU75VVFCX3H3F4DFJCULMMX7T2NH4ED3Q, seg_pos={0 0}, upload=713365]: Long tail cancellation due to optimal threshold Apr 2 00:01:26.729357871 (637h1m24.326128901s) [noise=true , address=193.149.164.66:4993 , piece_id=BZDCB4CN5ECUOJQBQKA73VTC6ZTT6DTNGFQYEP7ZVMNVPRCFBP2A, seg_pos={0 0}, upload=713365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bswx2GLREkFjLdeGYptnwDdcmm2FdUiJHycqMbimfXBT139va): protocol: expected piece hash; context canceled; EOF Apr 2 00:01:26.729365391 (637h1m24.326136414s) [noise=true , address=193.149.164.66:4993 , piece_id=BZDCB4CN5ECUOJQBQKA73VTC6ZTT6DTNGFQYEP7ZVMNVPRCFBP2A, seg_pos={0 0}, upload=713365]: Long tail cancellation due to optimal threshold Apr 2 00:01:26.729800455 (637h1m24.326571479s) [noise=true , address=145.79.227.13:4826 , piece_id=ABKYY2FG4CT3LNCRP2IKFVUVY6IJLUMCXRXHPMUVCLX2AXZNGGEQ, seg_pos={0 0}, upload=713365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548): protocol: expected piece hash; context canceled; EOF Apr 2 00:01:26.729809669 (637h1m24.326580692s) [noise=true , address=145.79.227.13:4826 , piece_id=ABKYY2FG4CT3LNCRP2IKFVUVY6IJLUMCXRXHPMUVCLX2AXZNGGEQ, seg_pos={0 0}, upload=713365]: Long tail cancellation due to optimal threshold Apr 2 00:01:26.729859653 (637h1m24.326630667s) [seg_pos={0 0}, upload=713365]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 2 00:01:26.730089202 (637h1m24.326860227s) [upload=713365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:01:26.730097415 (637h1m24.326868445s) [upload=713365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:04:22.648262973 (637h4m20.245034007s) [upload=713365]: Done uploading segments... Apr 2 00:04:22.648395374 (637h4m20.245166408s) [upload=713365]: Upload finished. err: metaclient: context deadline exceeded Duration: 2m56.592383756s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..................................................XXXXXXXXXX] duration: 5.001472532s trace: https://jaeger.dev.storj.io/trace/225098543822fb0e FAIL (2026-04-02 00:03:59 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001499193s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [.....................................................XXXXXX.] duration: 25.366417063s trace: https://jaeger.dev.storj.io/trace/6a7bd19464145af2 FAIL (2026-04-02 00:03:47 UTC) operation: Commit ======ERROR====== uplink: metaclient: internal error ======LOGS====== Apr 2 00:02:52.098779856 (637h2m49.69555089s) [upload=713388]: Starting upload Apr 2 00:02:52.098870776 (637h2m49.69564179s) [upload=713388]: Done starting upload Apr 2 00:02:52.099009562 (637h2m49.695780585s) [upload=713388]: Uploading segments... Apr 2 00:02:52.099172380 (637h2m49.695943412s) [upload=713388]: Got next segment. Inline: true Apr 2 00:02:52.099180066 (637h2m49.695951095s) [upload=713388]: Waiting for error group managing segments... Apr 2 00:02:52.099264917 (637h2m49.696035951s) [upload=713388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:02:52.099272729 (637h2m49.696043763s) [upload=713388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Apr 2 00:02:52.099290320 (637h2m49.696061361s) [upload=713388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:03:22.115657878 (637h3m19.712428918s) [upload=713388]: Done uploading segments... Apr 2 00:03:22.115722556 (637h3m19.71249368s) [upload=713388]: Upload finished. err: metaclient: internal error Duration:* 25.366434359s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 464.943106ms trace: https://jaeger.dev.storj.io/trace/3c28cdfe17d1a942 FAIL (2026-04-02 00:03:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:02:37.851070386 (637h2m35.44784141s) [upload=713386]: Starting upload Apr 2 00:02:37.851166355 (637h2m35.447937379s) [upload=713386]: Done starting upload Apr 2 00:02:37.851253090 (637h2m35.44802412s) [upload=713386]: Uploading segments... Apr 2 00:02:37.851793181 (637h2m35.448564196s) [upload=713386]: Got next segment. Inline: false Apr 2 00:02:37.851882102 (637h2m35.448653125s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:02:37.851889461 (637h2m35.448660484s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:02:37.880067208 (637h2m35.476838238s) [seg_pos={0 0}, upload=713386]: Begin upload segment... Apr 2 00:02:37.880168602 (637h2m35.476939634s) [seg_pos={0 0}, upload=713386]: Done begin upload segment. Apr 2 00:02:37.880207987 (637h2m35.476979021s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880214330 (637h2m35.476985339s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880292505 (637h2m35.477063531s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881027652 (637h2m35.477798688s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881378064 (637h2m35.478149089s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882097992 (637h2m35.478869026s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882586634 (637h2m35.479357658s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.883643537 (637h2m35.48041457s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884145002 (637h2m35.480916034s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884766674 (637h2m35.481537743s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885263298 (637h2m35.482034334s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885763219 (637h2m35.482534252s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886298227 (637h2m35.483069261s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886892344 (637h2m35.483663375s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.887518235 (637h2m35.484289271s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.888216955 (637h2m35.484987987s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.976004543 (637h2m35.572775567s) [upload=713386]: Next returned nil segment Apr 2 00:02:37.976025238 (637h2m35.572796273s) [upload=713386]: Waiting for error group managing segments... Apr 2 00:02:37.982357614 (637h2m35.579128639s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.982449885 (637h2m35.5792209s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983162285 (637h2m35.579933315s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983404595 (637h2m35.580175625s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983964188 (637h2m35.580735217s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.984025323 (637h2m35.58079634s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.985346526 (637h2m35.582117555s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986081391 (637h2m35.582852422s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986551623 (637h2m35.583322653s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.987396144 (637h2m35.584167177s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.997416565 (637h2m35.594187596s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005002081 (637h2m35.601773128s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005136838 (637h2m35.601907854s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005157484 (637h2m35.601928499s) [seg_pos={0 0}, upload=713386]: Satisfied cohort requirements after 13 pieces Apr 2 00:02:38.005386492 (637h2m35.602157525s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005396281 (637h2m35.60216731s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005531231 (637h2m35.602302263s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005570651 (637h2m35.602341684s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005692951 (637h2m35.602463983s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005700918 (637h2m35.602471942s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005722537 (637h2m35.60249356s) [seg_pos={0 0}, upload=713386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 2 00:02:38.005805596 (637h2m35.60257662s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:02:38.005812420 (637h2m35.602583444s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:03:07.848626111 (637h3m5.445397142s) [upload=713386]: Done uploading segments... Apr 2 00:03:07.848811089 (637h3m5.445582113s) [upload=713386]: Upload finished. err: metaclient: context deadline exceeded Duration: 465.062954ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...................................................XXXXXXXXX] duration: 5.000851799s trace: https://jaeger.dev.storj.io/trace/15eb2a3a7f7e80b5 FAIL (2026-04-02 00:03:29 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000954048s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 6 out of 60 is failed (threshold=3) [......................................................XXXXXX] duration: 30.053915498s trace: https://jaeger.dev.storj.io/trace/6a7bd19464145af2 FAIL (2026-04-02 00:03:22 UTC) operation: Commit ======ERROR====== uplink: metaclient: internal error ======LOGS====== Apr 2 00:02:52.098779856 (637h2m49.69555089s) [upload=713388]: Starting upload Apr 2 00:02:52.098870776 (637h2m49.69564179s) [upload=713388]: Done starting upload Apr 2 00:02:52.099009562 (637h2m49.695780585s) [upload=713388]: Uploading segments... Apr 2 00:02:52.099172380 (637h2m49.695943412s) [upload=713388]: Got next segment. Inline: true Apr 2 00:02:52.099180066 (637h2m49.695951095s) [upload=713388]: Waiting for error group managing segments... Apr 2 00:02:52.099264917 (637h2m49.696035951s) [upload=713388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:02:52.099272729 (637h2m49.696043763s) [upload=713388]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Apr 2 00:02:52.099290320 (637h2m49.696061361s) [upload=713388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:03:22.115657878 (637h3m19.712428918s) [upload=713388]: Done uploading segments... Apr 2 00:03:22.115722556 (637h3m19.71249368s) [upload=713388]: Upload finished. err: metaclient: internal error Duration:* 30.053930001s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.036984596s trace: https://jaeger.dev.storj.io/trace/3c28cdfe17d1a942 FAIL (2026-04-02 00:03:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:02:37.851070386 (637h2m35.44784141s) [upload=713386]: Starting upload Apr 2 00:02:37.851166355 (637h2m35.447937379s) [upload=713386]: Done starting upload Apr 2 00:02:37.851253090 (637h2m35.44802412s) [upload=713386]: Uploading segments... Apr 2 00:02:37.851793181 (637h2m35.448564196s) [upload=713386]: Got next segment. Inline: false Apr 2 00:02:37.851882102 (637h2m35.448653125s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:02:37.851889461 (637h2m35.448660484s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:02:37.880067208 (637h2m35.476838238s) [seg_pos={0 0}, upload=713386]: Begin upload segment... Apr 2 00:02:37.880168602 (637h2m35.476939634s) [seg_pos={0 0}, upload=713386]: Done begin upload segment. Apr 2 00:02:37.880207987 (637h2m35.476979021s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880214330 (637h2m35.476985339s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880292505 (637h2m35.477063531s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881027652 (637h2m35.477798688s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881378064 (637h2m35.478149089s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882097992 (637h2m35.478869026s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882586634 (637h2m35.479357658s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.883643537 (637h2m35.48041457s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884145002 (637h2m35.480916034s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884766674 (637h2m35.481537743s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885263298 (637h2m35.482034334s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885763219 (637h2m35.482534252s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886298227 (637h2m35.483069261s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886892344 (637h2m35.483663375s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.887518235 (637h2m35.484289271s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.888216955 (637h2m35.484987987s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.976004543 (637h2m35.572775567s) [upload=713386]: Next returned nil segment Apr 2 00:02:37.976025238 (637h2m35.572796273s) [upload=713386]: Waiting for error group managing segments... Apr 2 00:02:37.982357614 (637h2m35.579128639s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.982449885 (637h2m35.5792209s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983162285 (637h2m35.579933315s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983404595 (637h2m35.580175625s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983964188 (637h2m35.580735217s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.984025323 (637h2m35.58079634s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.985346526 (637h2m35.582117555s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986081391 (637h2m35.582852422s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986551623 (637h2m35.583322653s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.987396144 (637h2m35.584167177s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.997416565 (637h2m35.594187596s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005002081 (637h2m35.601773128s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005136838 (637h2m35.601907854s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005157484 (637h2m35.601928499s) [seg_pos={0 0}, upload=713386]: Satisfied cohort requirements after 13 pieces Apr 2 00:02:38.005386492 (637h2m35.602157525s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005396281 (637h2m35.60216731s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005531231 (637h2m35.602302263s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005570651 (637h2m35.602341684s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005692951 (637h2m35.602463983s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005700918 (637h2m35.602471942s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005722537 (637h2m35.60249356s) [seg_pos={0 0}, upload=713386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 2 00:02:38.005805596 (637h2m35.60257662s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:02:38.005812420 (637h2m35.602583444s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:03:07.848626111 (637h3m5.445397142s) [upload=713386]: Done uploading segments... Apr 2 00:03:07.848811089 (637h3m5.445582113s) [upload=713386]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.037011097s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [....................................................XXXXXXXX] duration: 5.000713812s trace: https://jaeger.dev.storj.io/trace/4451a27bd19c9cd4 FAIL (2026-04-02 00:03:01 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00073981s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 1m0.001520353s trace: https://jaeger.dev.storj.io/trace/576d9f32eb7314c3 FAIL (2026-04-02 00:02:51 UTC) operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:01:52.023779533 (637h1m49.620550559s) [upload=713371]: Starting upload Apr 2 00:01:52.023848579 (637h1m49.620619607s) [upload=713371]: Done starting upload Apr 2 00:01:52.024086546 (637h1m49.620857559s) [upload=713371]: Uploading segments... Apr 2 00:01:52.024156226 (637h1m49.620927252s) [upload=713371]: Got next segment. Inline: true Apr 2 00:01:52.024163084 (637h1m49.620934107s) [upload=713371]: Waiting for error group managing segments... Apr 2 00:01:52.024249994 (637h1m49.621021007s) [upload=713371]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:01:52.024271992 (637h1m49.621043018s) [upload=713371]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Apr 2 00:01:52.024278727 (637h1m49.62104975s) [upload=713371]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:02:09.781748127 (637h2m7.378519161s) [upload=713371]: Done uploading segments... Apr 2 00:02:09.781775582 (637h2m7.378546605s) [upload=713371]: Upload finished. err: <nil> Duration:* 1m0.001536783s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 1m0.001720519s trace: https://jaeger.dev.storj.io/trace/2fd4f11d98d1311e FAIL (2026-04-02 00:01:51 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:00:52.015581429 (637h0m49.612352456s) [upload=713354]: Starting upload Apr 2 00:00:52.015648175 (637h0m49.612419208s) [upload=713354]: Done starting upload Apr 2 00:00:52.015749646 (637h0m49.612520672s) [upload=713354]: Uploading segments... Apr 2 00:00:52.015815611 (637h0m49.612586644s) [upload=713354]: Got next segment. Inline: true Apr 2 00:00:52.015821982 (637h0m49.612593006s) [upload=713354]: Waiting for error group managing segments... Apr 2 00:00:52.015874049 (637h0m49.612645076s) [upload=713354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:00:52.015880132 (637h0m49.612651156s) [upload=713354]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Apr 2 00:00:52.015884212 (637h0m49.612655236s) [upload=713354]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:01:51.924401792 (637h1m49.521172826s) [upload=713354]: Done uploading segments... Apr 2 00:01:51.924448735 (637h1m49.521219767s) [upload=713354]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1m0.001736518s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.....................................................XXXXXXX] duration: 5.002954064s trace: https://jaeger.dev.storj.io/trace/7f815c3c2622b7b8 FAIL (2026-04-02 00:01:48 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.003117124s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................................XX.X] duration: 1m16.857785734s trace: https://jaeger.dev.storj.io/trace/1810c169e569b510 FAIL (2026-04-02 00:01:25 UTC) operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:00:09.578178265 (637h0m7.17494948s) [upload=713341]: Starting upload Apr 2 00:00:09.578250095 (637h0m7.175021132s) [upload=713341]: Done starting upload Apr 2 00:00:09.578609226 (637h0m7.175380247s) [upload=713341]: Uploading segments... Apr 2 00:00:09.578672897 (637h0m7.175443932s) [upload=713341]: Got next segment. Inline: false Apr 2 00:00:09.580208537 (637h0m7.176979565s) [upload=713341]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:00:09.580226034 (637h0m7.176997067s) [upload=713341]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:00:09.631244883 (637h0m7.228015917s) [seg_pos={0 0}, upload=713341]: Begin upload segment... Apr 2 00:00:09.631324543 (637h0m7.228095568s) [seg_pos={0 0}, upload=713341]: Done begin upload segment. Apr 2 00:00:09.631367336 (637h0m7.228138369s) [noise=true , address=45.91.116.6:4982 , piece_id=OA6YZE4XPK2Z7QHUKX56RXGFEOCBVSFQRD46NIAZG4A3GDRCEGOA, seg_pos={0 0}, upload=713341]: Uploading piece... Apr 2 00:00:09.632103680 (637h0m7.228874877s) [noise=true , address=145.79.227.22:4530 , piece_id=Z3THQBNKSQZ6U2EFM3G5O3XER7NURLJIEV37B7SZFPWDGZ5WR5MA, seg_pos={0 0}, upload=713341]: Uploading piece... Apr 2 00:00:09.633121465 (637h0m7.2298925s) [noise=true , address=167.17.72.68:4670 , piece_id=DJWRLFERFXBI2EIPFMQBFATDKERBCJTXGXII3RTO32NB3JAHGOEA, seg_pos={0 0}, upload=713341]: Uploading piece... Apr 2 00:00:09.633706138 (637h0m7.230477172s) [noise=true , address=167.17.72.70:4772 , piece_id=2KB63XAD3YUDS7OAWM6E7VHBBCNRIXWMXI5S4TUYMLWHYTARYIRA, seg_pos={0 0}, upload=713341]: Uploading piece... Apr 2 00:00:09.635095025 (637h0m7.231866061s) [noise=true , address=167.17.72.69:4725 , piece_id=BKDRU6FOLPQFLLCJ5SCC4H2ZNSAZUMBFDGVX4EJGPLYP7GV7A35A, seg_pos={0 0}, upload=713341]: Uploading piece... Apr 2 00:00:09.635725486 (637h0m7.232496516s) [noise=true , address=193.149.164.66:4981 , piece_id=V4JF5DACXPAZAMRU7LAJKXY2ZCIMMMQWNBYBMUQ5I3ULMI2C3EHA, seg_pos={0 0}, upload=713341]: Uploading piece... Apr 2 00:00:09.636178154 (637h0m7.232949187s) [noise=true , address=193.149.164.143:4703 , piece_id=6GEQSR3WEAA7GSYBPD2YTT23HTV4EVEDYQML6PLUXPI3OHCKNJGQ, seg_pos={0 0}, upload=713341]: Uploading piece... Apr 2 00:00:09.636637583 (637h0m7.233408614s) [noise=true , address=193.149.164.70:4119 , piece_id=H2NJMEJZA5QNU5UQ7WGF4MSTGGQI4KEZKHOS7MRKPJEWTM2RH3EA, seg_pos={0 0}, upload=713341]: Uploading piece... Apr 2 00:00:09.637106802 (637h0m7.233877813s) [noise=true , address=193.149.164.67:4007 , piece_id=OVG74ALOJERXYWLFLZAQUTHZSJPM5BSEXLCPYZJXJZGUQFWUKIOA, seg_pos={0 0}, upload=713341]: Uploading piece... Apr 2 00:00:09.637679953 (637h0m7.23445099s) [noise=true , address=145.79.227.18:4034 , piece_id=MKAV2LPGJQSQIFHG4IHTH46JO5VJG7PYRHCEVXXXZB272FBSCJTQ, seg_pos={0 0}, upload=713341]: Uploading piece... Apr 2 00:00:09.637781002 (637h0m7.234552033s) [noise=true , address=77.90.31.19:4867 , piece_id=H2CSUIALMZISBI72XTRTELIUKZEQI3WDKDSIRTTHHBXF4JRO5PJA, seg_pos={0 0}, upload=713341]: Uploading piece... Apr 2 00:00:09.638274340 (637h0m7.235045374s) [noise=true , address=64.49.13.137:4503 , piece_id=KR6KHI3VSO5WC2AFA6CCQUWB2VON5JQ5Y6LTCDZ7AWNK6OS4ZYKA, seg_pos={0 0}, upload=713341]: Uploading piece... Apr 2 00:00:09.638604465 (637h0m7.235375502s) [noise=true , address=145.79.227.17:4003 , piece_id=LN4WBWW4OMCRAFIGAHQ7VUNISDDAS2VJPBZLYWUH73SANSHZ5X2A, seg_pos={0 0}, upload=713341]: Uploading piece... Apr 2 00:00:09.638821504 (637h0m7.235592538s) [noise=true , address=193.149.164.68:4042 , piece_id=QPJRY3YOOZXEFAA6TY2IUGWMGZEUTF7YRBWZVZP4T6BNXA3IBBJA, seg_pos={0 0}, upload=713341]: Uploading piece... Apr 2 00:00:09.639344926 (637h0m7.236115959s) [noise=true , address=145.79.227.10:4433 , piece_id=EXZGZDMP6FATE44TCWXRUIVXXVXQOIEPCPRSMP2XDRVKDB7CF4UA, seg_pos={0 0}, upload=713341]: Uploading piece... Apr 2 00:00:09.639847910 (637h0m7.236618944s) [noise=true , address=77.90.31.12:4620 , piece_id=J6YNJKU4YEB7VVGPZGGQQYDOOWOME25LNH33S7DYPJTIZEUM3JKQ, seg_pos={0 0}, upload=713341]: Uploading piece... Apr 2 00:00:09.641619902 (637h0m7.23839095s) [noise=true , address=64.49.13.139:4469 , piece_id=2HOXACKUB5QM3YDFS4WISZ2I4R72MO6N5DFIAJPXC7456WG3Z3WQ, seg_pos={0 0}, upload=713341]: Uploading piece... Apr 2 00:00:09.642175660 (637h0m7.238946697s) [noise=true , address=145.79.227.11:4743 , piece_id=K6QMZFZ4OW3JMDIE5MN232BFDDMZMUW6IE35I742JML4QPJGDMJA, seg_pos={0 0}, upload=713341]: Uploading piece... Apr 2 00:00:09.729243730 (637h0m7.326014744s) [upload=713341]: Next returned nil segment Apr 2 00:00:09.729257514 (637h0m7.326028529s) [upload=713341]: Waiting for error group managing segments... Apr 2 00:00:09.737493491 (637h0m7.334264529s) [noise=true , address=145.79.227.10:4433 , piece_id=EXZGZDMP6FATE44TCWXRUIVXXVXQOIEPCPRSMP2XDRVKDB7CF4UA, seg_pos={0 0}, upload=713341]: Done uploading piece. err: <nil> Apr 2 00:00:09.740931300 (637h0m7.337702339s) [noise=true , address=145.79.227.11:4743 , piece_id=K6QMZFZ4OW3JMDIE5MN232BFDDMZMUW6IE35I742JML4QPJGDMJA, seg_pos={0 0}, upload=713341]: Done uploading piece. err: <nil> Apr 2 00:00:09.741013342 (637h0m7.337784375s) [noise=true , address=145.79.227.17:4003 , piece_id=LN4WBWW4OMCRAFIGAHQ7VUNISDDAS2VJPBZLYWUH73SANSHZ5X2A, seg_pos={0 0}, upload=713341]: Done uploading piece. err: <nil> Apr 2 00:00:09.751559871 (637h0m7.34833091s) [noise=true , address=77.90.31.19:4867 , piece_id=H2CSUIALMZISBI72XTRTELIUKZEQI3WDKDSIRTTHHBXF4JRO5PJA, seg_pos={0 0}, upload=713341]: Done uploading piece. err: <nil> Apr 2 00:00:09.751621827 (637h0m7.348392842s) [noise=true , address=45.91.116.6:4982 , piece_id=OA6YZE4XPK2Z7QHUKX56RXGFEOCBVSFQRD46NIAZG4A3GDRCEGOA, seg_pos={0 0}, upload=713341]: Done uploading piece. err: <nil> Apr 2 00:00:09.756526772 (637h0m7.353297798s) [noise=true , address=145.79.227.18:4034 , piece_id=MKAV2LPGJQSQIFHG4IHTH46JO5VJG7PYRHCEVXXXZB272FBSCJTQ, seg_pos={0 0}, upload=713341]: Done uploading piece. err: <nil> Apr 2 00:00:09.757280921 (637h0m7.354051949s) [noise=true , address=145.79.227.22:4530 , piece_id=Z3THQBNKSQZ6U2EFM3G5O3XER7NURLJIEV37B7SZFPWDGZ5WR5MA, seg_pos={0 0}, upload=713341]: Done uploading piece. err: <nil> Apr 2 00:00:09.757616039 (637h0m7.354387064s) [noise=true , address=77.90.31.12:4620 , piece_id=J6YNJKU4YEB7VVGPZGGQQYDOOWOME25LNH33S7DYPJTIZEUM3JKQ, seg_pos={0 0}, upload=713341]: Done uploading piece. err: <nil> Apr 2 00:00:10.034981246 (637h0m7.631752284s) [noise=true , address=167.17.72.68:4670 , piece_id=DJWRLFERFXBI2EIPFMQBFATDKERBCJTXGXII3RTO32NB3JAHGOEA, seg_pos={0 0}, upload=713341]: Done uploading piece. err: <nil> Apr 2 00:00:10.076942219 (637h0m7.673713253s) [noise=true , address=167.17.72.70:4772 , piece_id=2KB63XAD3YUDS7OAWM6E7VHBBCNRIXWMXI5S4TUYMLWHYTARYIRA, seg_pos={0 0}, upload=713341]: Done uploading piece. err: <nil> Apr 2 00:00:10.085818190 (637h0m7.682589224s) [noise=true , address=167.17.72.69:4725 , piece_id=BKDRU6FOLPQFLLCJ5SCC4H2ZNSAZUMBFDGVX4EJGPLYP7GV7A35A, seg_pos={0 0}, upload=713341]: Done uploading piece. err: <nil> Apr 2 00:00:10.278444905 (637h0m7.87521593s) [noise=true , address=64.49.13.139:4469 , piece_id=2HOXACKUB5QM3YDFS4WISZ2I4R72MO6N5DFIAJPXC7456WG3Z3WQ, seg_pos={0 0}, upload=713341]: Done uploading piece. err: <nil> Apr 2 00:00:10.278515316 (637h0m7.875286343s) [noise=true , address=193.149.164.66:4981 , piece_id=V4JF5DACXPAZAMRU7LAJKXY2ZCIMMMQWNBYBMUQ5I3ULMI2C3EHA, seg_pos={0 0}, upload=713341]: Done uploading piece. err: <nil> Apr 2 00:00:10.279409260 (637h0m7.87618027s) [noise=true , address=193.149.164.70:4119 , piece_id=H2NJMEJZA5QNU5UQ7WGF4MSTGGQI4KEZKHOS7MRKPJEWTM2RH3EA, seg_pos={0 0}, upload=713341]: Done uploading piece. err: <nil> Apr 2 00:00:10.279433241 (637h0m7.876204275s) [seg_pos={0 0}, upload=713341]: Satisfied cohort requirements after 14 pieces Apr 2 00:00:10.279859032 (637h0m7.876630068s) [noise=true , address=64.49.13.137:4503 , piece_id=KR6KHI3VSO5WC2AFA6CCQUWB2VON5JQ5Y6LTCDZ7AWNK6OS4ZYKA, seg_pos={0 0}, upload=713341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iDPmJ6m5ARuB918CAzwhuLm9pgCw6Ci1giQBaQJ9rD8ruEMcq): protocol: expected piece hash; context canceled; EOF Apr 2 00:00:10.279873615 (637h0m7.87664465s) [noise=true , address=64.49.13.137:4503 , piece_id=KR6KHI3VSO5WC2AFA6CCQUWB2VON5JQ5Y6LTCDZ7AWNK6OS4ZYKA, seg_pos={0 0}, upload=713341]: Long tail cancellation due to optimal threshold Apr 2 00:00:10.279896125 (637h0m7.876667159s) [noise=true , address=193.149.164.143:4703 , piece_id=6GEQSR3WEAA7GSYBPD2YTT23HTV4EVEDYQML6PLUXPI3OHCKNJGQ, seg_pos={0 0}, upload=713341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eomwcuHK58QuxBp3sAARJjm6Doc7zEzRYCodsTzFT4v6Nti9g): protocol: expected piece hash; context canceled; EOF Apr 2 00:00:10.279912290 (637h0m7.876683326s) [noise=true , address=193.149.164.143:4703 , piece_id=6GEQSR3WEAA7GSYBPD2YTT23HTV4EVEDYQML6PLUXPI3OHCKNJGQ, seg_pos={0 0}, upload=713341]: Long tail cancellation due to optimal threshold Apr 2 00:00:10.280320156 (637h0m7.877091191s) [noise=true , address=193.149.164.68:4042 , piece_id=QPJRY3YOOZXEFAA6TY2IUGWMGZEUTF7YRBWZVZP4T6BNXA3IBBJA, seg_pos={0 0}, upload=713341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehZDmABRtxFvCaSXpNMQ4r5LfVmxgDpg5xeAbYNm63BpdN6p4): protocol: expected piece hash; context canceled; EOF Apr 2 00:00:10.280334013 (637h0m7.877105051s) [noise=true , address=193.149.164.68:4042 , piece_id=QPJRY3YOOZXEFAA6TY2IUGWMGZEUTF7YRBWZVZP4T6BNXA3IBBJA, seg_pos={0 0}, upload=713341]: Long tail cancellation due to optimal threshold Apr 2 00:00:10.280348668 (637h0m7.877119704s) [noise=true , address=193.149.164.67:4007 , piece_id=OVG74ALOJERXYWLFLZAQUTHZSJPM5BSEXLCPYZJXJZGUQFWUKIOA, seg_pos={0 0}, upload=713341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQPp9ETa2PjVWiRzjoqQC1NF8cuuftT4mvij54fA4YMY9deDy): protocol: expected piece hash; context canceled; EOF Apr 2 00:00:10.280362545 (637h0m7.87713358s) [noise=true , address=193.149.164.67:4007 , piece_id=OVG74ALOJERXYWLFLZAQUTHZSJPM5BSEXLCPYZJXJZGUQFWUKIOA, seg_pos={0 0}, upload=713341]: Long tail cancellation due to optimal threshold Apr 2 00:00:10.280392534 (637h0m7.877163548s) [seg_pos={0 0}, upload=713341]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 2 00:00:10.280511751 (637h0m7.877282786s) [upload=713341]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:00:10.280521186 (637h0m7.877292224s) [upload=713341]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:01:25.848155700 (637h1m23.444926751s) [upload=713341]: Done uploading segments... Apr 2 00:01:25.848322306 (637h1m23.445093341s) [upload=713341]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m16.857819577s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [......................................................XXXXXX] duration: 5.001000318s trace: https://jaeger.dev.storj.io/trace/3f2198d6faa6afd5 FAIL (2026-04-02 00:01:01 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001050739s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 1m0.002200987s trace: https://jaeger.dev.storj.io/trace/1f6a2921bab9a6de FAIL (2026-04-02 00:00:51 UTC) operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 1 23:59:51.798806233 (636h59m49.395577265s) [upload=713334]: Starting upload Apr 1 23:59:51.798900025 (636h59m49.39567107s) [upload=713334]: Done starting upload Apr 1 23:59:51.799043455 (636h59m49.395814471s) [upload=713334]: Uploading segments... Apr 1 23:59:51.799154274 (636h59m49.395925326s) [upload=713334]: Got next segment. Inline: true Apr 1 23:59:51.799165362 (636h59m49.395936415s) [upload=713334]: Waiting for error group managing segments... Apr 1 23:59:51.799249096 (636h59m49.396020123s) [upload=713334]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 1 23:59:51.799257450 (636h59m49.39602848s) [upload=713334]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Apr 1 23:59:51.799262236 (636h59m49.39603326s) [upload=713334]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:00:24.419949789 (637h0m22.016720823s) [upload=713334]: Done uploading segments... Apr 2 00:00:24.419977228 (637h0m22.016748264s) [upload=713334]: Upload finished. err: <nil> Duration:* 1m0.002242607s core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)