Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX......................] duration: 848.016687ms trace: https://jaeger.dev.storj.io/trace/7766953e0b42057d FAIL (2026-04-02 00:39:57 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 848.044127ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 565.091311ms trace: https://jaeger.dev.storj.io/trace/3c28cdfe17d1a942 FAIL (2026-04-02 00:39:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:02:37.851070386 (637h2m35.44784141s) [upload=713386]: Starting upload Apr 2 00:02:37.851166355 (637h2m35.447937379s) [upload=713386]: Done starting upload Apr 2 00:02:37.851253090 (637h2m35.44802412s) [upload=713386]: Uploading segments... Apr 2 00:02:37.851793181 (637h2m35.448564196s) [upload=713386]: Got next segment. Inline: false Apr 2 00:02:37.851882102 (637h2m35.448653125s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:02:37.851889461 (637h2m35.448660484s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:02:37.880067208 (637h2m35.476838238s) [seg_pos={0 0}, upload=713386]: Begin upload segment... Apr 2 00:02:37.880168602 (637h2m35.476939634s) [seg_pos={0 0}, upload=713386]: Done begin upload segment. Apr 2 00:02:37.880207987 (637h2m35.476979021s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880214330 (637h2m35.476985339s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880292505 (637h2m35.477063531s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881027652 (637h2m35.477798688s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881378064 (637h2m35.478149089s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882097992 (637h2m35.478869026s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882586634 (637h2m35.479357658s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.883643537 (637h2m35.48041457s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884145002 (637h2m35.480916034s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884766674 (637h2m35.481537743s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885263298 (637h2m35.482034334s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885763219 (637h2m35.482534252s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886298227 (637h2m35.483069261s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886892344 (637h2m35.483663375s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.887518235 (637h2m35.484289271s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.888216955 (637h2m35.484987987s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.976004543 (637h2m35.572775567s) [upload=713386]: Next returned nil segment Apr 2 00:02:37.976025238 (637h2m35.572796273s) [upload=713386]: Waiting for error group managing segments... Apr 2 00:02:37.982357614 (637h2m35.579128639s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.982449885 (637h2m35.5792209s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983162285 (637h2m35.579933315s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983404595 (637h2m35.580175625s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983964188 (637h2m35.580735217s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.984025323 (637h2m35.58079634s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.985346526 (637h2m35.582117555s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986081391 (637h2m35.582852422s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986551623 (637h2m35.583322653s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.987396144 (637h2m35.584167177s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.997416565 (637h2m35.594187596s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005002081 (637h2m35.601773128s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005136838 (637h2m35.601907854s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005157484 (637h2m35.601928499s) [seg_pos={0 0}, upload=713386]: Satisfied cohort requirements after 13 pieces Apr 2 00:02:38.005386492 (637h2m35.602157525s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005396281 (637h2m35.60216731s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005531231 (637h2m35.602302263s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005570651 (637h2m35.602341684s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005692951 (637h2m35.602463983s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005700918 (637h2m35.602471942s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005722537 (637h2m35.60249356s) [seg_pos={0 0}, upload=713386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 2 00:02:38.005805596 (637h2m35.60257662s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:02:38.005812420 (637h2m35.602583444s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:03:07.848626111 (637h3m5.445397142s) [upload=713386]: Done uploading segments... Apr 2 00:03:07.848811089 (637h3m5.445582113s) [upload=713386]: Upload finished. err: metaclient: context deadline exceeded Duration: 565.126433ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 10 out of 60 is failed (threshold=3) [.......XXXXXX.XXX........X..................................] duration: 641.251291ms trace: https://jaeger.dev.storj.io/trace/2d309a7a06fb1cdc FAIL (2026-04-02 00:39:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:12:21.530762169 (637h12m19.127533179s) [upload=713560]: Starting upload Apr 2 00:12:21.530842522 (637h12m19.127613558s) [upload=713560]: Done starting upload Apr 2 00:12:21.530951258 (637h12m19.127722284s) [upload=713560]: Uploading segments... Apr 2 00:12:21.531087313 (637h12m19.127858578s) [upload=713560]: Got next segment. Inline: true Apr 2 00:12:21.531098521 (637h12m19.127869553s) [upload=713560]: Waiting for error group managing segments... Apr 2 00:12:21.531173202 (637h12m19.127944216s) [upload=713560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:12:21.531178995 (637h12m19.127950018s) [upload=713560]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Apr 2 00:12:21.531183386 (637h12m19.12795441s) [upload=713560]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:13:21.491464871 (637h13m19.088235905s) [upload=713560]: Done uploading segments... Apr 2 00:13:21.491522533 (637h13m19.088293572s) [upload=713560]: Upload finished. err: metaclient: context deadline exceeded Duration:* 641.289794ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........XX.XXX...X........................................] duration: 1.166873621s trace: https://jaeger.dev.storj.io/trace/7b66886c63182d02 FAIL (2026-04-02 00:39:21 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:08:32.201995061 (637h8m29.798766091s) [upload=713490]: Starting upload Apr 2 00:08:32.202058979 (637h8m29.798830011s) [upload=713490]: Done starting upload Apr 2 00:08:32.202104136 (637h8m29.798875153s) [upload=713490]: Uploading segments... Apr 2 00:08:32.202468215 (637h8m29.799239227s) [upload=713490]: Got next segment. Inline: false Apr 2 00:08:32.202545730 (637h8m29.799316756s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:08:32.202554537 (637h8m29.799325554s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:08:32.236884387 (637h8m29.83365542s) [seg_pos={0 0}, upload=713490]: Begin upload segment... Apr 2 00:08:32.236972300 (637h8m29.833743327s) [seg_pos={0 0}, upload=713490]: Done begin upload segment. Apr 2 00:08:32.237040287 (637h8m29.833811319s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237609577 (637h8m29.834380609s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237761417 (637h8m29.834532443s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.238323459 (637h8m29.835094484s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239091520 (637h8m29.835862549s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239476832 (637h8m29.836247858s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240185307 (637h8m29.836956336s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240585259 (637h8m29.837356286s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241120411 (637h8m29.83789144s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241483943 (637h8m29.838254967s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241758194 (637h8m29.838529219s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.242223914 (637h8m29.838994946s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243144784 (637h8m29.839915821s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243610121 (637h8m29.840381134s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244322341 (637h8m29.841093371s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244793070 (637h8m29.841564104s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245167641 (637h8m29.841938667s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245545293 (637h8m29.842316322s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.309369659 (637h8m29.9061407s) [upload=713490]: Next returned nil segment Apr 2 00:08:32.309386955 (637h8m29.906157989s) [upload=713490]: Waiting for error group managing segments... Apr 2 00:08:32.314998992 (637h8m29.911770022s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.317608657 (637h8m29.914379694s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.331442646 (637h8m29.92821367s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.334324786 (637h8m29.931095819s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.347843004 (637h8m29.944614014s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.349783033 (637h8m29.946554099s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.449314364 (637h8m30.04608539s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.466602858 (637h8m30.063373882s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.620540338 (637h8m30.217311348s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.627266502 (637h8m30.224037516s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.633186725 (637h8m30.229957751s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.836095281 (637h8m30.432866306s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.841319166 (637h8m30.438090195s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852957995 (637h8m30.449729018s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852982095 (637h8m30.449753118s) [seg_pos={0 0}, upload=713490]: Satisfied cohort requirements after 14 pieces Apr 2 00:08:32.853223544 (637h8m30.449994576s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853232550 (637h8m30.450003573s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853611136 (637h8m30.450382165s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MVWRaM417kW63ffaSvNzgtk1PxPQbbvDAxEzFSLVjDkhkLt7f): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853619602 (637h8m30.450390625s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853821693 (637h8m30.450592723s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GihGm5Lo1EDs1ZMqctuqxN3VVpvZHz7iBo4aSvhRhF2sBkAKx): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853828723 (637h8m30.450599745s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853975694 (637h8m30.450746723s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853981928 (637h8m30.450752951s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.854004392 (637h8m30.450775415s) [seg_pos={0 0}, upload=713490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 2 00:08:32.854137807 (637h8m30.450908831s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:08:32.854144625 (637h8m30.450915648s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:08:32.919468065 (637h8m30.516239079s) [upload=713490]: Done uploading segments... Apr 2 00:08:32.919489033 (637h8m30.516260054s) [upload=713490]: Upload finished. err: <nil> Duration: 1.166909689s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.....................] duration: 599.547618ms trace: https://jaeger.dev.storj.io/trace/7766953e0b42057d FAIL (2026-04-02 00:39:09 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 599.583715ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........XX.XXX...X.......................................] duration: 2.475352673s trace: https://jaeger.dev.storj.io/trace/7b66886c63182d02 FAIL (2026-04-02 00:38:56 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:08:32.201995061 (637h8m29.798766091s) [upload=713490]: Starting upload Apr 2 00:08:32.202058979 (637h8m29.798830011s) [upload=713490]: Done starting upload Apr 2 00:08:32.202104136 (637h8m29.798875153s) [upload=713490]: Uploading segments... Apr 2 00:08:32.202468215 (637h8m29.799239227s) [upload=713490]: Got next segment. Inline: false Apr 2 00:08:32.202545730 (637h8m29.799316756s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:08:32.202554537 (637h8m29.799325554s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:08:32.236884387 (637h8m29.83365542s) [seg_pos={0 0}, upload=713490]: Begin upload segment... Apr 2 00:08:32.236972300 (637h8m29.833743327s) [seg_pos={0 0}, upload=713490]: Done begin upload segment. Apr 2 00:08:32.237040287 (637h8m29.833811319s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237609577 (637h8m29.834380609s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237761417 (637h8m29.834532443s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.238323459 (637h8m29.835094484s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239091520 (637h8m29.835862549s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239476832 (637h8m29.836247858s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240185307 (637h8m29.836956336s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240585259 (637h8m29.837356286s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241120411 (637h8m29.83789144s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241483943 (637h8m29.838254967s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241758194 (637h8m29.838529219s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.242223914 (637h8m29.838994946s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243144784 (637h8m29.839915821s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243610121 (637h8m29.840381134s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244322341 (637h8m29.841093371s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244793070 (637h8m29.841564104s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245167641 (637h8m29.841938667s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245545293 (637h8m29.842316322s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.309369659 (637h8m29.9061407s) [upload=713490]: Next returned nil segment Apr 2 00:08:32.309386955 (637h8m29.906157989s) [upload=713490]: Waiting for error group managing segments... Apr 2 00:08:32.314998992 (637h8m29.911770022s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.317608657 (637h8m29.914379694s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.331442646 (637h8m29.92821367s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.334324786 (637h8m29.931095819s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.347843004 (637h8m29.944614014s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.349783033 (637h8m29.946554099s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.449314364 (637h8m30.04608539s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.466602858 (637h8m30.063373882s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.620540338 (637h8m30.217311348s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.627266502 (637h8m30.224037516s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.633186725 (637h8m30.229957751s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.836095281 (637h8m30.432866306s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.841319166 (637h8m30.438090195s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852957995 (637h8m30.449729018s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852982095 (637h8m30.449753118s) [seg_pos={0 0}, upload=713490]: Satisfied cohort requirements after 14 pieces Apr 2 00:08:32.853223544 (637h8m30.449994576s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853232550 (637h8m30.450003573s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853611136 (637h8m30.450382165s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MVWRaM417kW63ffaSvNzgtk1PxPQbbvDAxEzFSLVjDkhkLt7f): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853619602 (637h8m30.450390625s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853821693 (637h8m30.450592723s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GihGm5Lo1EDs1ZMqctuqxN3VVpvZHz7iBo4aSvhRhF2sBkAKx): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853828723 (637h8m30.450599745s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853975694 (637h8m30.450746723s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853981928 (637h8m30.450752951s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.854004392 (637h8m30.450775415s) [seg_pos={0 0}, upload=713490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 2 00:08:32.854137807 (637h8m30.450908831s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:08:32.854144625 (637h8m30.450915648s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:08:32.919468065 (637h8m30.516239079s) [upload=713490]: Done uploading segments... Apr 2 00:08:32.919489033 (637h8m30.516260054s) [upload=713490]: Upload finished. err: <nil> Duration: 2.475373719s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [....XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX....................] duration: 573.875761ms trace: https://jaeger.dev.storj.io/trace/7766953e0b42057d FAIL (2026-04-02 00:38:42 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 573.905577ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 487.438237ms trace: https://jaeger.dev.storj.io/trace/3c28cdfe17d1a942 FAIL (2026-04-02 00:38:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:02:37.851070386 (637h2m35.44784141s) [upload=713386]: Starting upload Apr 2 00:02:37.851166355 (637h2m35.447937379s) [upload=713386]: Done starting upload Apr 2 00:02:37.851253090 (637h2m35.44802412s) [upload=713386]: Uploading segments... Apr 2 00:02:37.851793181 (637h2m35.448564196s) [upload=713386]: Got next segment. Inline: false Apr 2 00:02:37.851882102 (637h2m35.448653125s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:02:37.851889461 (637h2m35.448660484s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:02:37.880067208 (637h2m35.476838238s) [seg_pos={0 0}, upload=713386]: Begin upload segment... Apr 2 00:02:37.880168602 (637h2m35.476939634s) [seg_pos={0 0}, upload=713386]: Done begin upload segment. Apr 2 00:02:37.880207987 (637h2m35.476979021s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880214330 (637h2m35.476985339s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880292505 (637h2m35.477063531s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881027652 (637h2m35.477798688s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881378064 (637h2m35.478149089s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882097992 (637h2m35.478869026s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882586634 (637h2m35.479357658s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.883643537 (637h2m35.48041457s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884145002 (637h2m35.480916034s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884766674 (637h2m35.481537743s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885263298 (637h2m35.482034334s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885763219 (637h2m35.482534252s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886298227 (637h2m35.483069261s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886892344 (637h2m35.483663375s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.887518235 (637h2m35.484289271s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.888216955 (637h2m35.484987987s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.976004543 (637h2m35.572775567s) [upload=713386]: Next returned nil segment Apr 2 00:02:37.976025238 (637h2m35.572796273s) [upload=713386]: Waiting for error group managing segments... Apr 2 00:02:37.982357614 (637h2m35.579128639s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.982449885 (637h2m35.5792209s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983162285 (637h2m35.579933315s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983404595 (637h2m35.580175625s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983964188 (637h2m35.580735217s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.984025323 (637h2m35.58079634s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.985346526 (637h2m35.582117555s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986081391 (637h2m35.582852422s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986551623 (637h2m35.583322653s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.987396144 (637h2m35.584167177s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.997416565 (637h2m35.594187596s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005002081 (637h2m35.601773128s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005136838 (637h2m35.601907854s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005157484 (637h2m35.601928499s) [seg_pos={0 0}, upload=713386]: Satisfied cohort requirements after 13 pieces Apr 2 00:02:38.005386492 (637h2m35.602157525s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005396281 (637h2m35.60216731s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005531231 (637h2m35.602302263s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005570651 (637h2m35.602341684s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005692951 (637h2m35.602463983s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005700918 (637h2m35.602471942s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005722537 (637h2m35.60249356s) [seg_pos={0 0}, upload=713386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 2 00:02:38.005805596 (637h2m35.60257662s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:02:38.005812420 (637h2m35.602583444s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:03:07.848626111 (637h3m5.445397142s) [upload=713386]: Done uploading segments... Apr 2 00:03:07.848811089 (637h3m5.445582113s) [upload=713386]: Upload finished. err: metaclient: context deadline exceeded Duration: 487.462932ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [............XX.XXX...X......................................] duration: 1.217507241s trace: https://jaeger.dev.storj.io/trace/7b66886c63182d02 FAIL (2026-04-02 00:38:21 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:08:32.201995061 (637h8m29.798766091s) [upload=713490]: Starting upload Apr 2 00:08:32.202058979 (637h8m29.798830011s) [upload=713490]: Done starting upload Apr 2 00:08:32.202104136 (637h8m29.798875153s) [upload=713490]: Uploading segments... Apr 2 00:08:32.202468215 (637h8m29.799239227s) [upload=713490]: Got next segment. Inline: false Apr 2 00:08:32.202545730 (637h8m29.799316756s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:08:32.202554537 (637h8m29.799325554s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:08:32.236884387 (637h8m29.83365542s) [seg_pos={0 0}, upload=713490]: Begin upload segment... Apr 2 00:08:32.236972300 (637h8m29.833743327s) [seg_pos={0 0}, upload=713490]: Done begin upload segment. Apr 2 00:08:32.237040287 (637h8m29.833811319s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237609577 (637h8m29.834380609s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237761417 (637h8m29.834532443s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.238323459 (637h8m29.835094484s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239091520 (637h8m29.835862549s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239476832 (637h8m29.836247858s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240185307 (637h8m29.836956336s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240585259 (637h8m29.837356286s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241120411 (637h8m29.83789144s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241483943 (637h8m29.838254967s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241758194 (637h8m29.838529219s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.242223914 (637h8m29.838994946s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243144784 (637h8m29.839915821s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243610121 (637h8m29.840381134s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244322341 (637h8m29.841093371s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244793070 (637h8m29.841564104s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245167641 (637h8m29.841938667s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245545293 (637h8m29.842316322s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.309369659 (637h8m29.9061407s) [upload=713490]: Next returned nil segment Apr 2 00:08:32.309386955 (637h8m29.906157989s) [upload=713490]: Waiting for error group managing segments... Apr 2 00:08:32.314998992 (637h8m29.911770022s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.317608657 (637h8m29.914379694s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.331442646 (637h8m29.92821367s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.334324786 (637h8m29.931095819s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.347843004 (637h8m29.944614014s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.349783033 (637h8m29.946554099s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.449314364 (637h8m30.04608539s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.466602858 (637h8m30.063373882s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.620540338 (637h8m30.217311348s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.627266502 (637h8m30.224037516s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.633186725 (637h8m30.229957751s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.836095281 (637h8m30.432866306s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.841319166 (637h8m30.438090195s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852957995 (637h8m30.449729018s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852982095 (637h8m30.449753118s) [seg_pos={0 0}, upload=713490]: Satisfied cohort requirements after 14 pieces Apr 2 00:08:32.853223544 (637h8m30.449994576s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853232550 (637h8m30.450003573s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853611136 (637h8m30.450382165s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MVWRaM417kW63ffaSvNzgtk1PxPQbbvDAxEzFSLVjDkhkLt7f): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853619602 (637h8m30.450390625s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853821693 (637h8m30.450592723s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GihGm5Lo1EDs1ZMqctuqxN3VVpvZHz7iBo4aSvhRhF2sBkAKx): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853828723 (637h8m30.450599745s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853975694 (637h8m30.450746723s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853981928 (637h8m30.450752951s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.854004392 (637h8m30.450775415s) [seg_pos={0 0}, upload=713490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 2 00:08:32.854137807 (637h8m30.450908831s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:08:32.854144625 (637h8m30.450915648s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:08:32.919468065 (637h8m30.516239079s) [upload=713490]: Done uploading segments... Apr 2 00:08:32.919489033 (637h8m30.516260054s) [upload=713490]: Upload finished. err: <nil> Duration: 1.217535083s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 10 out of 60 is failed (threshold=3) [........XXXXXX.XXX........X.................................] duration: 216.193564ms trace: https://jaeger.dev.storj.io/trace/2d309a7a06fb1cdc FAIL (2026-04-02 00:38:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:12:21.530762169 (637h12m19.127533179s) [upload=713560]: Starting upload Apr 2 00:12:21.530842522 (637h12m19.127613558s) [upload=713560]: Done starting upload Apr 2 00:12:21.530951258 (637h12m19.127722284s) [upload=713560]: Uploading segments... Apr 2 00:12:21.531087313 (637h12m19.127858578s) [upload=713560]: Got next segment. Inline: true Apr 2 00:12:21.531098521 (637h12m19.127869553s) [upload=713560]: Waiting for error group managing segments... Apr 2 00:12:21.531173202 (637h12m19.127944216s) [upload=713560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:12:21.531178995 (637h12m19.127950018s) [upload=713560]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Apr 2 00:12:21.531183386 (637h12m19.12795441s) [upload=713560]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:13:21.491464871 (637h13m19.088235905s) [upload=713560]: Done uploading segments... Apr 2 00:13:21.491522533 (637h13m19.088293572s) [upload=713560]: Upload finished. err: metaclient: context deadline exceeded Duration:* 216.216387ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 635.583162ms trace: https://jaeger.dev.storj.io/trace/3c28cdfe17d1a942 FAIL (2026-04-02 00:37:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:02:37.851070386 (637h2m35.44784141s) [upload=713386]: Starting upload Apr 2 00:02:37.851166355 (637h2m35.447937379s) [upload=713386]: Done starting upload Apr 2 00:02:37.851253090 (637h2m35.44802412s) [upload=713386]: Uploading segments... Apr 2 00:02:37.851793181 (637h2m35.448564196s) [upload=713386]: Got next segment. Inline: false Apr 2 00:02:37.851882102 (637h2m35.448653125s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:02:37.851889461 (637h2m35.448660484s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:02:37.880067208 (637h2m35.476838238s) [seg_pos={0 0}, upload=713386]: Begin upload segment... Apr 2 00:02:37.880168602 (637h2m35.476939634s) [seg_pos={0 0}, upload=713386]: Done begin upload segment. Apr 2 00:02:37.880207987 (637h2m35.476979021s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880214330 (637h2m35.476985339s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880292505 (637h2m35.477063531s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881027652 (637h2m35.477798688s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881378064 (637h2m35.478149089s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882097992 (637h2m35.478869026s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882586634 (637h2m35.479357658s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.883643537 (637h2m35.48041457s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884145002 (637h2m35.480916034s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884766674 (637h2m35.481537743s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885263298 (637h2m35.482034334s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885763219 (637h2m35.482534252s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886298227 (637h2m35.483069261s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886892344 (637h2m35.483663375s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.887518235 (637h2m35.484289271s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.888216955 (637h2m35.484987987s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.976004543 (637h2m35.572775567s) [upload=713386]: Next returned nil segment Apr 2 00:02:37.976025238 (637h2m35.572796273s) [upload=713386]: Waiting for error group managing segments... Apr 2 00:02:37.982357614 (637h2m35.579128639s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.982449885 (637h2m35.5792209s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983162285 (637h2m35.579933315s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983404595 (637h2m35.580175625s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983964188 (637h2m35.580735217s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.984025323 (637h2m35.58079634s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.985346526 (637h2m35.582117555s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986081391 (637h2m35.582852422s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986551623 (637h2m35.583322653s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.987396144 (637h2m35.584167177s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.997416565 (637h2m35.594187596s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005002081 (637h2m35.601773128s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005136838 (637h2m35.601907854s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005157484 (637h2m35.601928499s) [seg_pos={0 0}, upload=713386]: Satisfied cohort requirements after 13 pieces Apr 2 00:02:38.005386492 (637h2m35.602157525s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005396281 (637h2m35.60216731s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005531231 (637h2m35.602302263s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005570651 (637h2m35.602341684s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005692951 (637h2m35.602463983s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005700918 (637h2m35.602471942s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005722537 (637h2m35.60249356s) [seg_pos={0 0}, upload=713386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 2 00:02:38.005805596 (637h2m35.60257662s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:02:38.005812420 (637h2m35.602583444s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:03:07.848626111 (637h3m5.445397142s) [upload=713386]: Done uploading segments... Apr 2 00:03:07.848811089 (637h3m5.445582113s) [upload=713386]: Upload finished. err: metaclient: context deadline exceeded Duration: 635.617221ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [.....XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...................] duration: 670.816766ms trace: https://jaeger.dev.storj.io/trace/7766953e0b42057d FAIL (2026-04-02 00:37:31 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 670.849816ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 10 out of 60 is failed (threshold=3) [.........XXXXXX.XXX........X................................] duration: 247.218275ms trace: https://jaeger.dev.storj.io/trace/2d309a7a06fb1cdc FAIL (2026-04-02 00:37:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:12:21.530762169 (637h12m19.127533179s) [upload=713560]: Starting upload Apr 2 00:12:21.530842522 (637h12m19.127613558s) [upload=713560]: Done starting upload Apr 2 00:12:21.530951258 (637h12m19.127722284s) [upload=713560]: Uploading segments... Apr 2 00:12:21.531087313 (637h12m19.127858578s) [upload=713560]: Got next segment. Inline: true Apr 2 00:12:21.531098521 (637h12m19.127869553s) [upload=713560]: Waiting for error group managing segments... Apr 2 00:12:21.531173202 (637h12m19.127944216s) [upload=713560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:12:21.531178995 (637h12m19.127950018s) [upload=713560]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Apr 2 00:12:21.531183386 (637h12m19.12795441s) [upload=713560]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:13:21.491464871 (637h13m19.088235905s) [upload=713560]: Done uploading segments... Apr 2 00:13:21.491522533 (637h13m19.088293572s) [upload=713560]: Upload finished. err: metaclient: context deadline exceeded Duration:* 247.289031ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............XX.XXX...X.....................................] duration: 1.126906282s trace: https://jaeger.dev.storj.io/trace/7b66886c63182d02 FAIL (2026-04-02 00:37:13 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:08:32.201995061 (637h8m29.798766091s) [upload=713490]: Starting upload Apr 2 00:08:32.202058979 (637h8m29.798830011s) [upload=713490]: Done starting upload Apr 2 00:08:32.202104136 (637h8m29.798875153s) [upload=713490]: Uploading segments... Apr 2 00:08:32.202468215 (637h8m29.799239227s) [upload=713490]: Got next segment. Inline: false Apr 2 00:08:32.202545730 (637h8m29.799316756s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:08:32.202554537 (637h8m29.799325554s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:08:32.236884387 (637h8m29.83365542s) [seg_pos={0 0}, upload=713490]: Begin upload segment... Apr 2 00:08:32.236972300 (637h8m29.833743327s) [seg_pos={0 0}, upload=713490]: Done begin upload segment. Apr 2 00:08:32.237040287 (637h8m29.833811319s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237609577 (637h8m29.834380609s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237761417 (637h8m29.834532443s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.238323459 (637h8m29.835094484s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239091520 (637h8m29.835862549s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239476832 (637h8m29.836247858s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240185307 (637h8m29.836956336s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240585259 (637h8m29.837356286s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241120411 (637h8m29.83789144s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241483943 (637h8m29.838254967s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241758194 (637h8m29.838529219s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.242223914 (637h8m29.838994946s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243144784 (637h8m29.839915821s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243610121 (637h8m29.840381134s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244322341 (637h8m29.841093371s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244793070 (637h8m29.841564104s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245167641 (637h8m29.841938667s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245545293 (637h8m29.842316322s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.309369659 (637h8m29.9061407s) [upload=713490]: Next returned nil segment Apr 2 00:08:32.309386955 (637h8m29.906157989s) [upload=713490]: Waiting for error group managing segments... Apr 2 00:08:32.314998992 (637h8m29.911770022s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.317608657 (637h8m29.914379694s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.331442646 (637h8m29.92821367s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.334324786 (637h8m29.931095819s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.347843004 (637h8m29.944614014s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.349783033 (637h8m29.946554099s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.449314364 (637h8m30.04608539s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.466602858 (637h8m30.063373882s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.620540338 (637h8m30.217311348s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.627266502 (637h8m30.224037516s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.633186725 (637h8m30.229957751s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.836095281 (637h8m30.432866306s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.841319166 (637h8m30.438090195s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852957995 (637h8m30.449729018s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852982095 (637h8m30.449753118s) [seg_pos={0 0}, upload=713490]: Satisfied cohort requirements after 14 pieces Apr 2 00:08:32.853223544 (637h8m30.449994576s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853232550 (637h8m30.450003573s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853611136 (637h8m30.450382165s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MVWRaM417kW63ffaSvNzgtk1PxPQbbvDAxEzFSLVjDkhkLt7f): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853619602 (637h8m30.450390625s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853821693 (637h8m30.450592723s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GihGm5Lo1EDs1ZMqctuqxN3VVpvZHz7iBo4aSvhRhF2sBkAKx): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853828723 (637h8m30.450599745s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853975694 (637h8m30.450746723s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853981928 (637h8m30.450752951s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.854004392 (637h8m30.450775415s) [seg_pos={0 0}, upload=713490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 2 00:08:32.854137807 (637h8m30.450908831s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:08:32.854144625 (637h8m30.450915648s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:08:32.919468065 (637h8m30.516239079s) [upload=713490]: Done uploading segments... Apr 2 00:08:32.919489033 (637h8m30.516260054s) [upload=713490]: Upload finished. err: <nil> Duration: 1.126935486s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 565.546386ms trace: https://jaeger.dev.storj.io/trace/3c28cdfe17d1a942 FAIL (2026-04-02 00:37:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:02:37.851070386 (637h2m35.44784141s) [upload=713386]: Starting upload Apr 2 00:02:37.851166355 (637h2m35.447937379s) [upload=713386]: Done starting upload Apr 2 00:02:37.851253090 (637h2m35.44802412s) [upload=713386]: Uploading segments... Apr 2 00:02:37.851793181 (637h2m35.448564196s) [upload=713386]: Got next segment. Inline: false Apr 2 00:02:37.851882102 (637h2m35.448653125s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:02:37.851889461 (637h2m35.448660484s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:02:37.880067208 (637h2m35.476838238s) [seg_pos={0 0}, upload=713386]: Begin upload segment... Apr 2 00:02:37.880168602 (637h2m35.476939634s) [seg_pos={0 0}, upload=713386]: Done begin upload segment. Apr 2 00:02:37.880207987 (637h2m35.476979021s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880214330 (637h2m35.476985339s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880292505 (637h2m35.477063531s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881027652 (637h2m35.477798688s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881378064 (637h2m35.478149089s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882097992 (637h2m35.478869026s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882586634 (637h2m35.479357658s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.883643537 (637h2m35.48041457s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884145002 (637h2m35.480916034s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884766674 (637h2m35.481537743s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885263298 (637h2m35.482034334s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885763219 (637h2m35.482534252s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886298227 (637h2m35.483069261s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886892344 (637h2m35.483663375s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.887518235 (637h2m35.484289271s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.888216955 (637h2m35.484987987s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.976004543 (637h2m35.572775567s) [upload=713386]: Next returned nil segment Apr 2 00:02:37.976025238 (637h2m35.572796273s) [upload=713386]: Waiting for error group managing segments... Apr 2 00:02:37.982357614 (637h2m35.579128639s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.982449885 (637h2m35.5792209s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983162285 (637h2m35.579933315s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983404595 (637h2m35.580175625s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983964188 (637h2m35.580735217s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.984025323 (637h2m35.58079634s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.985346526 (637h2m35.582117555s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986081391 (637h2m35.582852422s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986551623 (637h2m35.583322653s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.987396144 (637h2m35.584167177s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.997416565 (637h2m35.594187596s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005002081 (637h2m35.601773128s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005136838 (637h2m35.601907854s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005157484 (637h2m35.601928499s) [seg_pos={0 0}, upload=713386]: Satisfied cohort requirements after 13 pieces Apr 2 00:02:38.005386492 (637h2m35.602157525s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005396281 (637h2m35.60216731s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005531231 (637h2m35.602302263s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005570651 (637h2m35.602341684s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005692951 (637h2m35.602463983s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005700918 (637h2m35.602471942s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005722537 (637h2m35.60249356s) [seg_pos={0 0}, upload=713386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 2 00:02:38.005805596 (637h2m35.60257662s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:02:38.005812420 (637h2m35.602583444s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:03:07.848626111 (637h3m5.445397142s) [upload=713386]: Done uploading segments... Apr 2 00:03:07.848811089 (637h3m5.445582113s) [upload=713386]: Upload finished. err: metaclient: context deadline exceeded Duration: 565.718073ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..................] duration: 628.723645ms trace: https://jaeger.dev.storj.io/trace/7766953e0b42057d FAIL (2026-04-02 00:36:57 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 628.763667ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 10 out of 60 is failed (threshold=3) [..........XXXXXX.XXX........X...............................] duration: 276.635737ms trace: https://jaeger.dev.storj.io/trace/2d309a7a06fb1cdc FAIL (2026-04-02 00:36:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:12:21.530762169 (637h12m19.127533179s) [upload=713560]: Starting upload Apr 2 00:12:21.530842522 (637h12m19.127613558s) [upload=713560]: Done starting upload Apr 2 00:12:21.530951258 (637h12m19.127722284s) [upload=713560]: Uploading segments... Apr 2 00:12:21.531087313 (637h12m19.127858578s) [upload=713560]: Got next segment. Inline: true Apr 2 00:12:21.531098521 (637h12m19.127869553s) [upload=713560]: Waiting for error group managing segments... Apr 2 00:12:21.531173202 (637h12m19.127944216s) [upload=713560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:12:21.531178995 (637h12m19.127950018s) [upload=713560]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Apr 2 00:12:21.531183386 (637h12m19.12795441s) [upload=713560]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:13:21.491464871 (637h13m19.088235905s) [upload=713560]: Done uploading segments... Apr 2 00:13:21.491522533 (637h13m19.088293572s) [upload=713560]: Upload finished. err: metaclient: context deadline exceeded Duration:* 276.667297ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............XX.XXX...X....................................] duration: 1.225214655s trace: https://jaeger.dev.storj.io/trace/7b66886c63182d02 FAIL (2026-04-02 00:36:19 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:08:32.201995061 (637h8m29.798766091s) [upload=713490]: Starting upload Apr 2 00:08:32.202058979 (637h8m29.798830011s) [upload=713490]: Done starting upload Apr 2 00:08:32.202104136 (637h8m29.798875153s) [upload=713490]: Uploading segments... Apr 2 00:08:32.202468215 (637h8m29.799239227s) [upload=713490]: Got next segment. Inline: false Apr 2 00:08:32.202545730 (637h8m29.799316756s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:08:32.202554537 (637h8m29.799325554s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:08:32.236884387 (637h8m29.83365542s) [seg_pos={0 0}, upload=713490]: Begin upload segment... Apr 2 00:08:32.236972300 (637h8m29.833743327s) [seg_pos={0 0}, upload=713490]: Done begin upload segment. Apr 2 00:08:32.237040287 (637h8m29.833811319s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237609577 (637h8m29.834380609s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237761417 (637h8m29.834532443s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.238323459 (637h8m29.835094484s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239091520 (637h8m29.835862549s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239476832 (637h8m29.836247858s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240185307 (637h8m29.836956336s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240585259 (637h8m29.837356286s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241120411 (637h8m29.83789144s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241483943 (637h8m29.838254967s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241758194 (637h8m29.838529219s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.242223914 (637h8m29.838994946s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243144784 (637h8m29.839915821s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243610121 (637h8m29.840381134s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244322341 (637h8m29.841093371s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244793070 (637h8m29.841564104s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245167641 (637h8m29.841938667s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245545293 (637h8m29.842316322s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.309369659 (637h8m29.9061407s) [upload=713490]: Next returned nil segment Apr 2 00:08:32.309386955 (637h8m29.906157989s) [upload=713490]: Waiting for error group managing segments... Apr 2 00:08:32.314998992 (637h8m29.911770022s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.317608657 (637h8m29.914379694s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.331442646 (637h8m29.92821367s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.334324786 (637h8m29.931095819s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.347843004 (637h8m29.944614014s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.349783033 (637h8m29.946554099s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.449314364 (637h8m30.04608539s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.466602858 (637h8m30.063373882s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.620540338 (637h8m30.217311348s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.627266502 (637h8m30.224037516s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.633186725 (637h8m30.229957751s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.836095281 (637h8m30.432866306s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.841319166 (637h8m30.438090195s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852957995 (637h8m30.449729018s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852982095 (637h8m30.449753118s) [seg_pos={0 0}, upload=713490]: Satisfied cohort requirements after 14 pieces Apr 2 00:08:32.853223544 (637h8m30.449994576s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853232550 (637h8m30.450003573s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853611136 (637h8m30.450382165s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MVWRaM417kW63ffaSvNzgtk1PxPQbbvDAxEzFSLVjDkhkLt7f): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853619602 (637h8m30.450390625s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853821693 (637h8m30.450592723s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GihGm5Lo1EDs1ZMqctuqxN3VVpvZHz7iBo4aSvhRhF2sBkAKx): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853828723 (637h8m30.450599745s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853975694 (637h8m30.450746723s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853981928 (637h8m30.450752951s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.854004392 (637h8m30.450775415s) [seg_pos={0 0}, upload=713490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 2 00:08:32.854137807 (637h8m30.450908831s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:08:32.854144625 (637h8m30.450915648s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:08:32.919468065 (637h8m30.516239079s) [upload=713490]: Done uploading segments... Apr 2 00:08:32.919489033 (637h8m30.516260054s) [upload=713490]: Upload finished. err: <nil> Duration: 1.22531923s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 560.476585ms trace: https://jaeger.dev.storj.io/trace/3c28cdfe17d1a942 FAIL (2026-04-02 00:36:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:02:37.851070386 (637h2m35.44784141s) [upload=713386]: Starting upload Apr 2 00:02:37.851166355 (637h2m35.447937379s) [upload=713386]: Done starting upload Apr 2 00:02:37.851253090 (637h2m35.44802412s) [upload=713386]: Uploading segments... Apr 2 00:02:37.851793181 (637h2m35.448564196s) [upload=713386]: Got next segment. Inline: false Apr 2 00:02:37.851882102 (637h2m35.448653125s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:02:37.851889461 (637h2m35.448660484s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:02:37.880067208 (637h2m35.476838238s) [seg_pos={0 0}, upload=713386]: Begin upload segment... Apr 2 00:02:37.880168602 (637h2m35.476939634s) [seg_pos={0 0}, upload=713386]: Done begin upload segment. Apr 2 00:02:37.880207987 (637h2m35.476979021s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880214330 (637h2m35.476985339s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880292505 (637h2m35.477063531s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881027652 (637h2m35.477798688s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881378064 (637h2m35.478149089s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882097992 (637h2m35.478869026s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882586634 (637h2m35.479357658s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.883643537 (637h2m35.48041457s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884145002 (637h2m35.480916034s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884766674 (637h2m35.481537743s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885263298 (637h2m35.482034334s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885763219 (637h2m35.482534252s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886298227 (637h2m35.483069261s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886892344 (637h2m35.483663375s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.887518235 (637h2m35.484289271s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.888216955 (637h2m35.484987987s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.976004543 (637h2m35.572775567s) [upload=713386]: Next returned nil segment Apr 2 00:02:37.976025238 (637h2m35.572796273s) [upload=713386]: Waiting for error group managing segments... Apr 2 00:02:37.982357614 (637h2m35.579128639s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.982449885 (637h2m35.5792209s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983162285 (637h2m35.579933315s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983404595 (637h2m35.580175625s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983964188 (637h2m35.580735217s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.984025323 (637h2m35.58079634s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.985346526 (637h2m35.582117555s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986081391 (637h2m35.582852422s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986551623 (637h2m35.583322653s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.987396144 (637h2m35.584167177s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.997416565 (637h2m35.594187596s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005002081 (637h2m35.601773128s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005136838 (637h2m35.601907854s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005157484 (637h2m35.601928499s) [seg_pos={0 0}, upload=713386]: Satisfied cohort requirements after 13 pieces Apr 2 00:02:38.005386492 (637h2m35.602157525s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005396281 (637h2m35.60216731s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005531231 (637h2m35.602302263s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005570651 (637h2m35.602341684s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005692951 (637h2m35.602463983s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005700918 (637h2m35.602471942s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005722537 (637h2m35.60249356s) [seg_pos={0 0}, upload=713386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 2 00:02:38.005805596 (637h2m35.60257662s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:02:38.005812420 (637h2m35.602583444s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:03:07.848626111 (637h3m5.445397142s) [upload=713386]: Done uploading segments... Apr 2 00:03:07.848811089 (637h3m5.445582113s) [upload=713386]: Upload finished. err: metaclient: context deadline exceeded Duration: 560.500387ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 10 out of 60 is failed (threshold=3) [...........XXXXXX.XXX........X..............................] duration: 235.372525ms trace: https://jaeger.dev.storj.io/trace/2d309a7a06fb1cdc FAIL (2026-04-02 00:36:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:12:21.530762169 (637h12m19.127533179s) [upload=713560]: Starting upload Apr 2 00:12:21.530842522 (637h12m19.127613558s) [upload=713560]: Done starting upload Apr 2 00:12:21.530951258 (637h12m19.127722284s) [upload=713560]: Uploading segments... Apr 2 00:12:21.531087313 (637h12m19.127858578s) [upload=713560]: Got next segment. Inline: true Apr 2 00:12:21.531098521 (637h12m19.127869553s) [upload=713560]: Waiting for error group managing segments... Apr 2 00:12:21.531173202 (637h12m19.127944216s) [upload=713560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:12:21.531178995 (637h12m19.127950018s) [upload=713560]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Apr 2 00:12:21.531183386 (637h12m19.12795441s) [upload=713560]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:13:21.491464871 (637h13m19.088235905s) [upload=713560]: Done uploading segments... Apr 2 00:13:21.491522533 (637h13m19.088293572s) [upload=713560]: Upload finished. err: metaclient: context deadline exceeded Duration:* 235.45289ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.................] duration: 609.813743ms trace: https://jaeger.dev.storj.io/trace/7766953e0b42057d FAIL (2026-04-02 00:35:59 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 609.852358ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............XX.XXX...X...................................] duration: 1.437408262s trace: https://jaeger.dev.storj.io/trace/7b66886c63182d02 FAIL (2026-04-02 00:35:42 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:08:32.201995061 (637h8m29.798766091s) [upload=713490]: Starting upload Apr 2 00:08:32.202058979 (637h8m29.798830011s) [upload=713490]: Done starting upload Apr 2 00:08:32.202104136 (637h8m29.798875153s) [upload=713490]: Uploading segments... Apr 2 00:08:32.202468215 (637h8m29.799239227s) [upload=713490]: Got next segment. Inline: false Apr 2 00:08:32.202545730 (637h8m29.799316756s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:08:32.202554537 (637h8m29.799325554s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:08:32.236884387 (637h8m29.83365542s) [seg_pos={0 0}, upload=713490]: Begin upload segment... Apr 2 00:08:32.236972300 (637h8m29.833743327s) [seg_pos={0 0}, upload=713490]: Done begin upload segment. Apr 2 00:08:32.237040287 (637h8m29.833811319s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237609577 (637h8m29.834380609s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237761417 (637h8m29.834532443s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.238323459 (637h8m29.835094484s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239091520 (637h8m29.835862549s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239476832 (637h8m29.836247858s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240185307 (637h8m29.836956336s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240585259 (637h8m29.837356286s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241120411 (637h8m29.83789144s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241483943 (637h8m29.838254967s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241758194 (637h8m29.838529219s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.242223914 (637h8m29.838994946s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243144784 (637h8m29.839915821s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243610121 (637h8m29.840381134s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244322341 (637h8m29.841093371s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244793070 (637h8m29.841564104s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245167641 (637h8m29.841938667s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245545293 (637h8m29.842316322s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.309369659 (637h8m29.9061407s) [upload=713490]: Next returned nil segment Apr 2 00:08:32.309386955 (637h8m29.906157989s) [upload=713490]: Waiting for error group managing segments... Apr 2 00:08:32.314998992 (637h8m29.911770022s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.317608657 (637h8m29.914379694s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.331442646 (637h8m29.92821367s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.334324786 (637h8m29.931095819s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.347843004 (637h8m29.944614014s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.349783033 (637h8m29.946554099s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.449314364 (637h8m30.04608539s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.466602858 (637h8m30.063373882s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.620540338 (637h8m30.217311348s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.627266502 (637h8m30.224037516s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.633186725 (637h8m30.229957751s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.836095281 (637h8m30.432866306s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.841319166 (637h8m30.438090195s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852957995 (637h8m30.449729018s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852982095 (637h8m30.449753118s) [seg_pos={0 0}, upload=713490]: Satisfied cohort requirements after 14 pieces Apr 2 00:08:32.853223544 (637h8m30.449994576s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853232550 (637h8m30.450003573s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853611136 (637h8m30.450382165s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MVWRaM417kW63ffaSvNzgtk1PxPQbbvDAxEzFSLVjDkhkLt7f): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853619602 (637h8m30.450390625s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853821693 (637h8m30.450592723s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GihGm5Lo1EDs1ZMqctuqxN3VVpvZHz7iBo4aSvhRhF2sBkAKx): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853828723 (637h8m30.450599745s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853975694 (637h8m30.450746723s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853981928 (637h8m30.450752951s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.854004392 (637h8m30.450775415s) [seg_pos={0 0}, upload=713490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 2 00:08:32.854137807 (637h8m30.450908831s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:08:32.854144625 (637h8m30.450915648s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:08:32.919468065 (637h8m30.516239079s) [upload=713490]: Done uploading segments... Apr 2 00:08:32.919489033 (637h8m30.516260054s) [upload=713490]: Upload finished. err: <nil> Duration: 1.437437886s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 564.796245ms trace: https://jaeger.dev.storj.io/trace/3c28cdfe17d1a942 FAIL (2026-04-02 00:35:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:02:37.851070386 (637h2m35.44784141s) [upload=713386]: Starting upload Apr 2 00:02:37.851166355 (637h2m35.447937379s) [upload=713386]: Done starting upload Apr 2 00:02:37.851253090 (637h2m35.44802412s) [upload=713386]: Uploading segments... Apr 2 00:02:37.851793181 (637h2m35.448564196s) [upload=713386]: Got next segment. Inline: false Apr 2 00:02:37.851882102 (637h2m35.448653125s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:02:37.851889461 (637h2m35.448660484s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:02:37.880067208 (637h2m35.476838238s) [seg_pos={0 0}, upload=713386]: Begin upload segment... Apr 2 00:02:37.880168602 (637h2m35.476939634s) [seg_pos={0 0}, upload=713386]: Done begin upload segment. Apr 2 00:02:37.880207987 (637h2m35.476979021s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880214330 (637h2m35.476985339s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880292505 (637h2m35.477063531s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881027652 (637h2m35.477798688s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881378064 (637h2m35.478149089s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882097992 (637h2m35.478869026s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882586634 (637h2m35.479357658s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.883643537 (637h2m35.48041457s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884145002 (637h2m35.480916034s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884766674 (637h2m35.481537743s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885263298 (637h2m35.482034334s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885763219 (637h2m35.482534252s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886298227 (637h2m35.483069261s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886892344 (637h2m35.483663375s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.887518235 (637h2m35.484289271s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.888216955 (637h2m35.484987987s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.976004543 (637h2m35.572775567s) [upload=713386]: Next returned nil segment Apr 2 00:02:37.976025238 (637h2m35.572796273s) [upload=713386]: Waiting for error group managing segments... Apr 2 00:02:37.982357614 (637h2m35.579128639s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.982449885 (637h2m35.5792209s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983162285 (637h2m35.579933315s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983404595 (637h2m35.580175625s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983964188 (637h2m35.580735217s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.984025323 (637h2m35.58079634s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.985346526 (637h2m35.582117555s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986081391 (637h2m35.582852422s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986551623 (637h2m35.583322653s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.987396144 (637h2m35.584167177s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.997416565 (637h2m35.594187596s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005002081 (637h2m35.601773128s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005136838 (637h2m35.601907854s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005157484 (637h2m35.601928499s) [seg_pos={0 0}, upload=713386]: Satisfied cohort requirements after 13 pieces Apr 2 00:02:38.005386492 (637h2m35.602157525s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005396281 (637h2m35.60216731s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005531231 (637h2m35.602302263s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005570651 (637h2m35.602341684s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005692951 (637h2m35.602463983s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005700918 (637h2m35.602471942s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005722537 (637h2m35.60249356s) [seg_pos={0 0}, upload=713386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 2 00:02:38.005805596 (637h2m35.60257662s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:02:38.005812420 (637h2m35.602583444s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:03:07.848626111 (637h3m5.445397142s) [upload=713386]: Done uploading segments... Apr 2 00:03:07.848811089 (637h3m5.445582113s) [upload=713386]: Upload finished. err: metaclient: context deadline exceeded Duration: 564.929076ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX................] duration: 1.064320179s trace: https://jaeger.dev.storj.io/trace/7766953e0b42057d FAIL (2026-04-02 00:35:12 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.064350677s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 10 out of 60 is failed (threshold=3) [............XXXXXX.XXX........X.............................] duration: 336.925872ms trace: https://jaeger.dev.storj.io/trace/2d309a7a06fb1cdc FAIL (2026-04-02 00:34:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:12:21.530762169 (637h12m19.127533179s) [upload=713560]: Starting upload Apr 2 00:12:21.530842522 (637h12m19.127613558s) [upload=713560]: Done starting upload Apr 2 00:12:21.530951258 (637h12m19.127722284s) [upload=713560]: Uploading segments... Apr 2 00:12:21.531087313 (637h12m19.127858578s) [upload=713560]: Got next segment. Inline: true Apr 2 00:12:21.531098521 (637h12m19.127869553s) [upload=713560]: Waiting for error group managing segments... Apr 2 00:12:21.531173202 (637h12m19.127944216s) [upload=713560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:12:21.531178995 (637h12m19.127950018s) [upload=713560]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Apr 2 00:12:21.531183386 (637h12m19.12795441s) [upload=713560]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:13:21.491464871 (637h13m19.088235905s) [upload=713560]: Done uploading segments... Apr 2 00:13:21.491522533 (637h13m19.088293572s) [upload=713560]: Upload finished. err: metaclient: context deadline exceeded Duration:* 336.959356ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [................XX.XXX...X..................................] duration: 1.290214911s trace: https://jaeger.dev.storj.io/trace/7b66886c63182d02 FAIL (2026-04-02 00:34:53 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:08:32.201995061 (637h8m29.798766091s) [upload=713490]: Starting upload Apr 2 00:08:32.202058979 (637h8m29.798830011s) [upload=713490]: Done starting upload Apr 2 00:08:32.202104136 (637h8m29.798875153s) [upload=713490]: Uploading segments... Apr 2 00:08:32.202468215 (637h8m29.799239227s) [upload=713490]: Got next segment. Inline: false Apr 2 00:08:32.202545730 (637h8m29.799316756s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:08:32.202554537 (637h8m29.799325554s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:08:32.236884387 (637h8m29.83365542s) [seg_pos={0 0}, upload=713490]: Begin upload segment... Apr 2 00:08:32.236972300 (637h8m29.833743327s) [seg_pos={0 0}, upload=713490]: Done begin upload segment. Apr 2 00:08:32.237040287 (637h8m29.833811319s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237609577 (637h8m29.834380609s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237761417 (637h8m29.834532443s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.238323459 (637h8m29.835094484s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239091520 (637h8m29.835862549s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239476832 (637h8m29.836247858s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240185307 (637h8m29.836956336s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240585259 (637h8m29.837356286s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241120411 (637h8m29.83789144s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241483943 (637h8m29.838254967s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241758194 (637h8m29.838529219s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.242223914 (637h8m29.838994946s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243144784 (637h8m29.839915821s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243610121 (637h8m29.840381134s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244322341 (637h8m29.841093371s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244793070 (637h8m29.841564104s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245167641 (637h8m29.841938667s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245545293 (637h8m29.842316322s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.309369659 (637h8m29.9061407s) [upload=713490]: Next returned nil segment Apr 2 00:08:32.309386955 (637h8m29.906157989s) [upload=713490]: Waiting for error group managing segments... Apr 2 00:08:32.314998992 (637h8m29.911770022s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.317608657 (637h8m29.914379694s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.331442646 (637h8m29.92821367s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.334324786 (637h8m29.931095819s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.347843004 (637h8m29.944614014s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.349783033 (637h8m29.946554099s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.449314364 (637h8m30.04608539s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.466602858 (637h8m30.063373882s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.620540338 (637h8m30.217311348s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.627266502 (637h8m30.224037516s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.633186725 (637h8m30.229957751s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.836095281 (637h8m30.432866306s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.841319166 (637h8m30.438090195s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852957995 (637h8m30.449729018s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852982095 (637h8m30.449753118s) [seg_pos={0 0}, upload=713490]: Satisfied cohort requirements after 14 pieces Apr 2 00:08:32.853223544 (637h8m30.449994576s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853232550 (637h8m30.450003573s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853611136 (637h8m30.450382165s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MVWRaM417kW63ffaSvNzgtk1PxPQbbvDAxEzFSLVjDkhkLt7f): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853619602 (637h8m30.450390625s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853821693 (637h8m30.450592723s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GihGm5Lo1EDs1ZMqctuqxN3VVpvZHz7iBo4aSvhRhF2sBkAKx): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853828723 (637h8m30.450599745s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853975694 (637h8m30.450746723s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853981928 (637h8m30.450752951s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.854004392 (637h8m30.450775415s) [seg_pos={0 0}, upload=713490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 2 00:08:32.854137807 (637h8m30.450908831s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:08:32.854144625 (637h8m30.450915648s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:08:32.919468065 (637h8m30.516239079s) [upload=713490]: Done uploading segments... Apr 2 00:08:32.919489033 (637h8m30.516260054s) [upload=713490]: Upload finished. err: <nil> Duration: 1.290243043s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 559.827085ms trace: https://jaeger.dev.storj.io/trace/3c28cdfe17d1a942 FAIL (2026-04-02 00:34:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:02:37.851070386 (637h2m35.44784141s) [upload=713386]: Starting upload Apr 2 00:02:37.851166355 (637h2m35.447937379s) [upload=713386]: Done starting upload Apr 2 00:02:37.851253090 (637h2m35.44802412s) [upload=713386]: Uploading segments... Apr 2 00:02:37.851793181 (637h2m35.448564196s) [upload=713386]: Got next segment. Inline: false Apr 2 00:02:37.851882102 (637h2m35.448653125s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:02:37.851889461 (637h2m35.448660484s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:02:37.880067208 (637h2m35.476838238s) [seg_pos={0 0}, upload=713386]: Begin upload segment... Apr 2 00:02:37.880168602 (637h2m35.476939634s) [seg_pos={0 0}, upload=713386]: Done begin upload segment. Apr 2 00:02:37.880207987 (637h2m35.476979021s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880214330 (637h2m35.476985339s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880292505 (637h2m35.477063531s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881027652 (637h2m35.477798688s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881378064 (637h2m35.478149089s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882097992 (637h2m35.478869026s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882586634 (637h2m35.479357658s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.883643537 (637h2m35.48041457s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884145002 (637h2m35.480916034s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884766674 (637h2m35.481537743s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885263298 (637h2m35.482034334s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885763219 (637h2m35.482534252s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886298227 (637h2m35.483069261s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886892344 (637h2m35.483663375s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.887518235 (637h2m35.484289271s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.888216955 (637h2m35.484987987s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.976004543 (637h2m35.572775567s) [upload=713386]: Next returned nil segment Apr 2 00:02:37.976025238 (637h2m35.572796273s) [upload=713386]: Waiting for error group managing segments... Apr 2 00:02:37.982357614 (637h2m35.579128639s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.982449885 (637h2m35.5792209s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983162285 (637h2m35.579933315s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983404595 (637h2m35.580175625s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983964188 (637h2m35.580735217s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.984025323 (637h2m35.58079634s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.985346526 (637h2m35.582117555s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986081391 (637h2m35.582852422s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986551623 (637h2m35.583322653s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.987396144 (637h2m35.584167177s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.997416565 (637h2m35.594187596s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005002081 (637h2m35.601773128s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005136838 (637h2m35.601907854s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005157484 (637h2m35.601928499s) [seg_pos={0 0}, upload=713386]: Satisfied cohort requirements after 13 pieces Apr 2 00:02:38.005386492 (637h2m35.602157525s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005396281 (637h2m35.60216731s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005531231 (637h2m35.602302263s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005570651 (637h2m35.602341684s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005692951 (637h2m35.602463983s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005700918 (637h2m35.602471942s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005722537 (637h2m35.60249356s) [seg_pos={0 0}, upload=713386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 2 00:02:38.005805596 (637h2m35.60257662s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:02:38.005812420 (637h2m35.602583444s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:03:07.848626111 (637h3m5.445397142s) [upload=713386]: Done uploading segments... Apr 2 00:03:07.848811089 (637h3m5.445582113s) [upload=713386]: Upload finished. err: metaclient: context deadline exceeded Duration: 559.862791ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [.........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...............] duration: 1.122531698s trace: https://jaeger.dev.storj.io/trace/7766953e0b42057d FAIL (2026-04-02 00:34:27 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.122561368s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................XX.XXX...X.................................] duration: 2.280346069s trace: https://jaeger.dev.storj.io/trace/7b66886c63182d02 FAIL (2026-04-02 00:34:05 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:08:32.201995061 (637h8m29.798766091s) [upload=713490]: Starting upload Apr 2 00:08:32.202058979 (637h8m29.798830011s) [upload=713490]: Done starting upload Apr 2 00:08:32.202104136 (637h8m29.798875153s) [upload=713490]: Uploading segments... Apr 2 00:08:32.202468215 (637h8m29.799239227s) [upload=713490]: Got next segment. Inline: false Apr 2 00:08:32.202545730 (637h8m29.799316756s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:08:32.202554537 (637h8m29.799325554s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:08:32.236884387 (637h8m29.83365542s) [seg_pos={0 0}, upload=713490]: Begin upload segment... Apr 2 00:08:32.236972300 (637h8m29.833743327s) [seg_pos={0 0}, upload=713490]: Done begin upload segment. Apr 2 00:08:32.237040287 (637h8m29.833811319s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237609577 (637h8m29.834380609s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237761417 (637h8m29.834532443s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.238323459 (637h8m29.835094484s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239091520 (637h8m29.835862549s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239476832 (637h8m29.836247858s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240185307 (637h8m29.836956336s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240585259 (637h8m29.837356286s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241120411 (637h8m29.83789144s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241483943 (637h8m29.838254967s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241758194 (637h8m29.838529219s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.242223914 (637h8m29.838994946s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243144784 (637h8m29.839915821s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243610121 (637h8m29.840381134s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244322341 (637h8m29.841093371s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244793070 (637h8m29.841564104s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245167641 (637h8m29.841938667s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245545293 (637h8m29.842316322s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.309369659 (637h8m29.9061407s) [upload=713490]: Next returned nil segment Apr 2 00:08:32.309386955 (637h8m29.906157989s) [upload=713490]: Waiting for error group managing segments... Apr 2 00:08:32.314998992 (637h8m29.911770022s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.317608657 (637h8m29.914379694s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.331442646 (637h8m29.92821367s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.334324786 (637h8m29.931095819s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.347843004 (637h8m29.944614014s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.349783033 (637h8m29.946554099s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.449314364 (637h8m30.04608539s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.466602858 (637h8m30.063373882s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.620540338 (637h8m30.217311348s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.627266502 (637h8m30.224037516s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.633186725 (637h8m30.229957751s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.836095281 (637h8m30.432866306s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.841319166 (637h8m30.438090195s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852957995 (637h8m30.449729018s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852982095 (637h8m30.449753118s) [seg_pos={0 0}, upload=713490]: Satisfied cohort requirements after 14 pieces Apr 2 00:08:32.853223544 (637h8m30.449994576s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853232550 (637h8m30.450003573s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853611136 (637h8m30.450382165s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MVWRaM417kW63ffaSvNzgtk1PxPQbbvDAxEzFSLVjDkhkLt7f): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853619602 (637h8m30.450390625s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853821693 (637h8m30.450592723s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GihGm5Lo1EDs1ZMqctuqxN3VVpvZHz7iBo4aSvhRhF2sBkAKx): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853828723 (637h8m30.450599745s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853975694 (637h8m30.450746723s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853981928 (637h8m30.450752951s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.854004392 (637h8m30.450775415s) [seg_pos={0 0}, upload=713490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 2 00:08:32.854137807 (637h8m30.450908831s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:08:32.854144625 (637h8m30.450915648s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:08:32.919468065 (637h8m30.516239079s) [upload=713490]: Done uploading segments... Apr 2 00:08:32.919489033 (637h8m30.516260054s) [upload=713490]: Upload finished. err: <nil> Duration: 2.280381488s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 10 out of 60 is failed (threshold=3) [.............XXXXXX.XXX........X............................] duration: 248.752962ms trace: https://jaeger.dev.storj.io/trace/2d309a7a06fb1cdc FAIL (2026-04-02 00:33:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:12:21.530762169 (637h12m19.127533179s) [upload=713560]: Starting upload Apr 2 00:12:21.530842522 (637h12m19.127613558s) [upload=713560]: Done starting upload Apr 2 00:12:21.530951258 (637h12m19.127722284s) [upload=713560]: Uploading segments... Apr 2 00:12:21.531087313 (637h12m19.127858578s) [upload=713560]: Got next segment. Inline: true Apr 2 00:12:21.531098521 (637h12m19.127869553s) [upload=713560]: Waiting for error group managing segments... Apr 2 00:12:21.531173202 (637h12m19.127944216s) [upload=713560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:12:21.531178995 (637h12m19.127950018s) [upload=713560]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Apr 2 00:12:21.531183386 (637h12m19.12795441s) [upload=713560]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:13:21.491464871 (637h13m19.088235905s) [upload=713560]: Done uploading segments... Apr 2 00:13:21.491522533 (637h13m19.088293572s) [upload=713560]: Upload finished. err: metaclient: context deadline exceeded Duration:* 248.785348ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [..........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..............] duration: 652.673056ms trace: https://jaeger.dev.storj.io/trace/7766953e0b42057d FAIL (2026-04-02 00:33:29 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 652.70012ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................XX.XXX...X................................] duration: 1.194365815s trace: https://jaeger.dev.storj.io/trace/7b66886c63182d02 FAIL (2026-04-02 00:33:28 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:08:32.201995061 (637h8m29.798766091s) [upload=713490]: Starting upload Apr 2 00:08:32.202058979 (637h8m29.798830011s) [upload=713490]: Done starting upload Apr 2 00:08:32.202104136 (637h8m29.798875153s) [upload=713490]: Uploading segments... Apr 2 00:08:32.202468215 (637h8m29.799239227s) [upload=713490]: Got next segment. Inline: false Apr 2 00:08:32.202545730 (637h8m29.799316756s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:08:32.202554537 (637h8m29.799325554s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:08:32.236884387 (637h8m29.83365542s) [seg_pos={0 0}, upload=713490]: Begin upload segment... Apr 2 00:08:32.236972300 (637h8m29.833743327s) [seg_pos={0 0}, upload=713490]: Done begin upload segment. Apr 2 00:08:32.237040287 (637h8m29.833811319s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237609577 (637h8m29.834380609s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237761417 (637h8m29.834532443s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.238323459 (637h8m29.835094484s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239091520 (637h8m29.835862549s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239476832 (637h8m29.836247858s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240185307 (637h8m29.836956336s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240585259 (637h8m29.837356286s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241120411 (637h8m29.83789144s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241483943 (637h8m29.838254967s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241758194 (637h8m29.838529219s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.242223914 (637h8m29.838994946s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243144784 (637h8m29.839915821s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243610121 (637h8m29.840381134s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244322341 (637h8m29.841093371s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244793070 (637h8m29.841564104s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245167641 (637h8m29.841938667s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245545293 (637h8m29.842316322s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.309369659 (637h8m29.9061407s) [upload=713490]: Next returned nil segment Apr 2 00:08:32.309386955 (637h8m29.906157989s) [upload=713490]: Waiting for error group managing segments... Apr 2 00:08:32.314998992 (637h8m29.911770022s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.317608657 (637h8m29.914379694s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.331442646 (637h8m29.92821367s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.334324786 (637h8m29.931095819s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.347843004 (637h8m29.944614014s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.349783033 (637h8m29.946554099s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.449314364 (637h8m30.04608539s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.466602858 (637h8m30.063373882s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.620540338 (637h8m30.217311348s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.627266502 (637h8m30.224037516s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.633186725 (637h8m30.229957751s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.836095281 (637h8m30.432866306s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.841319166 (637h8m30.438090195s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852957995 (637h8m30.449729018s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852982095 (637h8m30.449753118s) [seg_pos={0 0}, upload=713490]: Satisfied cohort requirements after 14 pieces Apr 2 00:08:32.853223544 (637h8m30.449994576s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853232550 (637h8m30.450003573s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853611136 (637h8m30.450382165s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MVWRaM417kW63ffaSvNzgtk1PxPQbbvDAxEzFSLVjDkhkLt7f): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853619602 (637h8m30.450390625s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853821693 (637h8m30.450592723s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GihGm5Lo1EDs1ZMqctuqxN3VVpvZHz7iBo4aSvhRhF2sBkAKx): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853828723 (637h8m30.450599745s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853975694 (637h8m30.450746723s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853981928 (637h8m30.450752951s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.854004392 (637h8m30.450775415s) [seg_pos={0 0}, upload=713490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 2 00:08:32.854137807 (637h8m30.450908831s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:08:32.854144625 (637h8m30.450915648s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:08:32.919468065 (637h8m30.516239079s) [upload=713490]: Done uploading segments... Apr 2 00:08:32.919489033 (637h8m30.516260054s) [upload=713490]: Upload finished. err: <nil> Duration: 1.194418851s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 505.354198ms trace: https://jaeger.dev.storj.io/trace/3c28cdfe17d1a942 FAIL (2026-04-02 00:33:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:02:37.851070386 (637h2m35.44784141s) [upload=713386]: Starting upload Apr 2 00:02:37.851166355 (637h2m35.447937379s) [upload=713386]: Done starting upload Apr 2 00:02:37.851253090 (637h2m35.44802412s) [upload=713386]: Uploading segments... Apr 2 00:02:37.851793181 (637h2m35.448564196s) [upload=713386]: Got next segment. Inline: false Apr 2 00:02:37.851882102 (637h2m35.448653125s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:02:37.851889461 (637h2m35.448660484s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:02:37.880067208 (637h2m35.476838238s) [seg_pos={0 0}, upload=713386]: Begin upload segment... Apr 2 00:02:37.880168602 (637h2m35.476939634s) [seg_pos={0 0}, upload=713386]: Done begin upload segment. Apr 2 00:02:37.880207987 (637h2m35.476979021s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880214330 (637h2m35.476985339s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880292505 (637h2m35.477063531s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881027652 (637h2m35.477798688s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881378064 (637h2m35.478149089s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882097992 (637h2m35.478869026s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882586634 (637h2m35.479357658s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.883643537 (637h2m35.48041457s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884145002 (637h2m35.480916034s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884766674 (637h2m35.481537743s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885263298 (637h2m35.482034334s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885763219 (637h2m35.482534252s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886298227 (637h2m35.483069261s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886892344 (637h2m35.483663375s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.887518235 (637h2m35.484289271s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.888216955 (637h2m35.484987987s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.976004543 (637h2m35.572775567s) [upload=713386]: Next returned nil segment Apr 2 00:02:37.976025238 (637h2m35.572796273s) [upload=713386]: Waiting for error group managing segments... Apr 2 00:02:37.982357614 (637h2m35.579128639s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.982449885 (637h2m35.5792209s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983162285 (637h2m35.579933315s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983404595 (637h2m35.580175625s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983964188 (637h2m35.580735217s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.984025323 (637h2m35.58079634s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.985346526 (637h2m35.582117555s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986081391 (637h2m35.582852422s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986551623 (637h2m35.583322653s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.987396144 (637h2m35.584167177s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.997416565 (637h2m35.594187596s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005002081 (637h2m35.601773128s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005136838 (637h2m35.601907854s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005157484 (637h2m35.601928499s) [seg_pos={0 0}, upload=713386]: Satisfied cohort requirements after 13 pieces Apr 2 00:02:38.005386492 (637h2m35.602157525s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005396281 (637h2m35.60216731s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005531231 (637h2m35.602302263s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005570651 (637h2m35.602341684s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005692951 (637h2m35.602463983s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005700918 (637h2m35.602471942s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005722537 (637h2m35.60249356s) [seg_pos={0 0}, upload=713386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 2 00:02:38.005805596 (637h2m35.60257662s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:02:38.005812420 (637h2m35.602583444s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:03:07.848626111 (637h3m5.445397142s) [upload=713386]: Done uploading segments... Apr 2 00:03:07.848811089 (637h3m5.445582113s) [upload=713386]: Upload finished. err: metaclient: context deadline exceeded Duration: 505.398813ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 10 out of 60 is failed (threshold=3) [..............XXXXXX.XXX........X...........................] duration: 203.492668ms trace: https://jaeger.dev.storj.io/trace/2d309a7a06fb1cdc FAIL (2026-04-02 00:33:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:12:21.530762169 (637h12m19.127533179s) [upload=713560]: Starting upload Apr 2 00:12:21.530842522 (637h12m19.127613558s) [upload=713560]: Done starting upload Apr 2 00:12:21.530951258 (637h12m19.127722284s) [upload=713560]: Uploading segments... Apr 2 00:12:21.531087313 (637h12m19.127858578s) [upload=713560]: Got next segment. Inline: true Apr 2 00:12:21.531098521 (637h12m19.127869553s) [upload=713560]: Waiting for error group managing segments... Apr 2 00:12:21.531173202 (637h12m19.127944216s) [upload=713560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:12:21.531178995 (637h12m19.127950018s) [upload=713560]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Apr 2 00:12:21.531183386 (637h12m19.12795441s) [upload=713560]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:13:21.491464871 (637h13m19.088235905s) [upload=713560]: Done uploading segments... Apr 2 00:13:21.491522533 (637h13m19.088293572s) [upload=713560]: Upload finished. err: metaclient: context deadline exceeded Duration:* 203.519734ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................XX.XXX...X...............................] duration: 1.167580125s trace: https://jaeger.dev.storj.io/trace/7b66886c63182d02 FAIL (2026-04-02 00:32:59 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:08:32.201995061 (637h8m29.798766091s) [upload=713490]: Starting upload Apr 2 00:08:32.202058979 (637h8m29.798830011s) [upload=713490]: Done starting upload Apr 2 00:08:32.202104136 (637h8m29.798875153s) [upload=713490]: Uploading segments... Apr 2 00:08:32.202468215 (637h8m29.799239227s) [upload=713490]: Got next segment. Inline: false Apr 2 00:08:32.202545730 (637h8m29.799316756s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:08:32.202554537 (637h8m29.799325554s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:08:32.236884387 (637h8m29.83365542s) [seg_pos={0 0}, upload=713490]: Begin upload segment... Apr 2 00:08:32.236972300 (637h8m29.833743327s) [seg_pos={0 0}, upload=713490]: Done begin upload segment. Apr 2 00:08:32.237040287 (637h8m29.833811319s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237609577 (637h8m29.834380609s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237761417 (637h8m29.834532443s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.238323459 (637h8m29.835094484s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239091520 (637h8m29.835862549s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239476832 (637h8m29.836247858s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240185307 (637h8m29.836956336s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240585259 (637h8m29.837356286s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241120411 (637h8m29.83789144s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241483943 (637h8m29.838254967s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241758194 (637h8m29.838529219s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.242223914 (637h8m29.838994946s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243144784 (637h8m29.839915821s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243610121 (637h8m29.840381134s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244322341 (637h8m29.841093371s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244793070 (637h8m29.841564104s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245167641 (637h8m29.841938667s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245545293 (637h8m29.842316322s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.309369659 (637h8m29.9061407s) [upload=713490]: Next returned nil segment Apr 2 00:08:32.309386955 (637h8m29.906157989s) [upload=713490]: Waiting for error group managing segments... Apr 2 00:08:32.314998992 (637h8m29.911770022s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.317608657 (637h8m29.914379694s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.331442646 (637h8m29.92821367s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.334324786 (637h8m29.931095819s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.347843004 (637h8m29.944614014s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.349783033 (637h8m29.946554099s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.449314364 (637h8m30.04608539s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.466602858 (637h8m30.063373882s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.620540338 (637h8m30.217311348s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.627266502 (637h8m30.224037516s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.633186725 (637h8m30.229957751s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.836095281 (637h8m30.432866306s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.841319166 (637h8m30.438090195s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852957995 (637h8m30.449729018s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852982095 (637h8m30.449753118s) [seg_pos={0 0}, upload=713490]: Satisfied cohort requirements after 14 pieces Apr 2 00:08:32.853223544 (637h8m30.449994576s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853232550 (637h8m30.450003573s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853611136 (637h8m30.450382165s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MVWRaM417kW63ffaSvNzgtk1PxPQbbvDAxEzFSLVjDkhkLt7f): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853619602 (637h8m30.450390625s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853821693 (637h8m30.450592723s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GihGm5Lo1EDs1ZMqctuqxN3VVpvZHz7iBo4aSvhRhF2sBkAKx): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853828723 (637h8m30.450599745s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853975694 (637h8m30.450746723s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853981928 (637h8m30.450752951s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.854004392 (637h8m30.450775415s) [seg_pos={0 0}, upload=713490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 2 00:08:32.854137807 (637h8m30.450908831s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:08:32.854144625 (637h8m30.450915648s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:08:32.919468065 (637h8m30.516239079s) [upload=713490]: Done uploading segments... Apr 2 00:08:32.919489033 (637h8m30.516260054s) [upload=713490]: Upload finished. err: <nil> Duration: 1.167616571s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 458.204434ms trace: https://jaeger.dev.storj.io/trace/3c28cdfe17d1a942 FAIL (2026-04-02 00:32:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:02:37.851070386 (637h2m35.44784141s) [upload=713386]: Starting upload Apr 2 00:02:37.851166355 (637h2m35.447937379s) [upload=713386]: Done starting upload Apr 2 00:02:37.851253090 (637h2m35.44802412s) [upload=713386]: Uploading segments... Apr 2 00:02:37.851793181 (637h2m35.448564196s) [upload=713386]: Got next segment. Inline: false Apr 2 00:02:37.851882102 (637h2m35.448653125s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:02:37.851889461 (637h2m35.448660484s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:02:37.880067208 (637h2m35.476838238s) [seg_pos={0 0}, upload=713386]: Begin upload segment... Apr 2 00:02:37.880168602 (637h2m35.476939634s) [seg_pos={0 0}, upload=713386]: Done begin upload segment. Apr 2 00:02:37.880207987 (637h2m35.476979021s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880214330 (637h2m35.476985339s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880292505 (637h2m35.477063531s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881027652 (637h2m35.477798688s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881378064 (637h2m35.478149089s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882097992 (637h2m35.478869026s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882586634 (637h2m35.479357658s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.883643537 (637h2m35.48041457s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884145002 (637h2m35.480916034s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884766674 (637h2m35.481537743s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885263298 (637h2m35.482034334s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885763219 (637h2m35.482534252s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886298227 (637h2m35.483069261s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886892344 (637h2m35.483663375s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.887518235 (637h2m35.484289271s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.888216955 (637h2m35.484987987s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.976004543 (637h2m35.572775567s) [upload=713386]: Next returned nil segment Apr 2 00:02:37.976025238 (637h2m35.572796273s) [upload=713386]: Waiting for error group managing segments... Apr 2 00:02:37.982357614 (637h2m35.579128639s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.982449885 (637h2m35.5792209s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983162285 (637h2m35.579933315s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983404595 (637h2m35.580175625s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983964188 (637h2m35.580735217s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.984025323 (637h2m35.58079634s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.985346526 (637h2m35.582117555s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986081391 (637h2m35.582852422s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986551623 (637h2m35.583322653s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.987396144 (637h2m35.584167177s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.997416565 (637h2m35.594187596s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005002081 (637h2m35.601773128s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005136838 (637h2m35.601907854s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005157484 (637h2m35.601928499s) [seg_pos={0 0}, upload=713386]: Satisfied cohort requirements after 13 pieces Apr 2 00:02:38.005386492 (637h2m35.602157525s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005396281 (637h2m35.60216731s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005531231 (637h2m35.602302263s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005570651 (637h2m35.602341684s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005692951 (637h2m35.602463983s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005700918 (637h2m35.602471942s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005722537 (637h2m35.60249356s) [seg_pos={0 0}, upload=713386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 2 00:02:38.005805596 (637h2m35.60257662s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:02:38.005812420 (637h2m35.602583444s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:03:07.848626111 (637h3m5.445397142s) [upload=713386]: Done uploading segments... Apr 2 00:03:07.848811089 (637h3m5.445582113s) [upload=713386]: Upload finished. err: metaclient: context deadline exceeded Duration: 458.235726ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................XX.XXX...X..............................] duration: 1.228426833s trace: https://jaeger.dev.storj.io/trace/7b66886c63182d02 FAIL (2026-04-02 00:32:38 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:08:32.201995061 (637h8m29.798766091s) [upload=713490]: Starting upload Apr 2 00:08:32.202058979 (637h8m29.798830011s) [upload=713490]: Done starting upload Apr 2 00:08:32.202104136 (637h8m29.798875153s) [upload=713490]: Uploading segments... Apr 2 00:08:32.202468215 (637h8m29.799239227s) [upload=713490]: Got next segment. Inline: false Apr 2 00:08:32.202545730 (637h8m29.799316756s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:08:32.202554537 (637h8m29.799325554s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:08:32.236884387 (637h8m29.83365542s) [seg_pos={0 0}, upload=713490]: Begin upload segment... Apr 2 00:08:32.236972300 (637h8m29.833743327s) [seg_pos={0 0}, upload=713490]: Done begin upload segment. Apr 2 00:08:32.237040287 (637h8m29.833811319s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237609577 (637h8m29.834380609s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237761417 (637h8m29.834532443s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.238323459 (637h8m29.835094484s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239091520 (637h8m29.835862549s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239476832 (637h8m29.836247858s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240185307 (637h8m29.836956336s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240585259 (637h8m29.837356286s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241120411 (637h8m29.83789144s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241483943 (637h8m29.838254967s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241758194 (637h8m29.838529219s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.242223914 (637h8m29.838994946s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243144784 (637h8m29.839915821s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243610121 (637h8m29.840381134s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244322341 (637h8m29.841093371s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244793070 (637h8m29.841564104s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245167641 (637h8m29.841938667s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245545293 (637h8m29.842316322s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.309369659 (637h8m29.9061407s) [upload=713490]: Next returned nil segment Apr 2 00:08:32.309386955 (637h8m29.906157989s) [upload=713490]: Waiting for error group managing segments... Apr 2 00:08:32.314998992 (637h8m29.911770022s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.317608657 (637h8m29.914379694s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.331442646 (637h8m29.92821367s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.334324786 (637h8m29.931095819s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.347843004 (637h8m29.944614014s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.349783033 (637h8m29.946554099s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.449314364 (637h8m30.04608539s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.466602858 (637h8m30.063373882s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.620540338 (637h8m30.217311348s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.627266502 (637h8m30.224037516s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.633186725 (637h8m30.229957751s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.836095281 (637h8m30.432866306s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.841319166 (637h8m30.438090195s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852957995 (637h8m30.449729018s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852982095 (637h8m30.449753118s) [seg_pos={0 0}, upload=713490]: Satisfied cohort requirements after 14 pieces Apr 2 00:08:32.853223544 (637h8m30.449994576s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853232550 (637h8m30.450003573s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853611136 (637h8m30.450382165s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MVWRaM417kW63ffaSvNzgtk1PxPQbbvDAxEzFSLVjDkhkLt7f): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853619602 (637h8m30.450390625s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853821693 (637h8m30.450592723s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GihGm5Lo1EDs1ZMqctuqxN3VVpvZHz7iBo4aSvhRhF2sBkAKx): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853828723 (637h8m30.450599745s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853975694 (637h8m30.450746723s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853981928 (637h8m30.450752951s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.854004392 (637h8m30.450775415s) [seg_pos={0 0}, upload=713490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 2 00:08:32.854137807 (637h8m30.450908831s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:08:32.854144625 (637h8m30.450915648s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:08:32.919468065 (637h8m30.516239079s) [upload=713490]: Done uploading segments... Apr 2 00:08:32.919489033 (637h8m30.516260054s) [upload=713490]: Upload finished. err: <nil> Duration: 1.228466071s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [...........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.............] duration: 549.913729ms trace: https://jaeger.dev.storj.io/trace/7766953e0b42057d FAIL (2026-04-02 00:32:34 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 550.013954ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 491.847016ms trace: https://jaeger.dev.storj.io/trace/3c28cdfe17d1a942 FAIL (2026-04-02 00:32:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:02:37.851070386 (637h2m35.44784141s) [upload=713386]: Starting upload Apr 2 00:02:37.851166355 (637h2m35.447937379s) [upload=713386]: Done starting upload Apr 2 00:02:37.851253090 (637h2m35.44802412s) [upload=713386]: Uploading segments... Apr 2 00:02:37.851793181 (637h2m35.448564196s) [upload=713386]: Got next segment. Inline: false Apr 2 00:02:37.851882102 (637h2m35.448653125s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:02:37.851889461 (637h2m35.448660484s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:02:37.880067208 (637h2m35.476838238s) [seg_pos={0 0}, upload=713386]: Begin upload segment... Apr 2 00:02:37.880168602 (637h2m35.476939634s) [seg_pos={0 0}, upload=713386]: Done begin upload segment. Apr 2 00:02:37.880207987 (637h2m35.476979021s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880214330 (637h2m35.476985339s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880292505 (637h2m35.477063531s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881027652 (637h2m35.477798688s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881378064 (637h2m35.478149089s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882097992 (637h2m35.478869026s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882586634 (637h2m35.479357658s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.883643537 (637h2m35.48041457s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884145002 (637h2m35.480916034s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884766674 (637h2m35.481537743s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885263298 (637h2m35.482034334s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885763219 (637h2m35.482534252s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886298227 (637h2m35.483069261s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886892344 (637h2m35.483663375s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.887518235 (637h2m35.484289271s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.888216955 (637h2m35.484987987s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.976004543 (637h2m35.572775567s) [upload=713386]: Next returned nil segment Apr 2 00:02:37.976025238 (637h2m35.572796273s) [upload=713386]: Waiting for error group managing segments... Apr 2 00:02:37.982357614 (637h2m35.579128639s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.982449885 (637h2m35.5792209s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983162285 (637h2m35.579933315s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983404595 (637h2m35.580175625s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983964188 (637h2m35.580735217s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.984025323 (637h2m35.58079634s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.985346526 (637h2m35.582117555s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986081391 (637h2m35.582852422s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986551623 (637h2m35.583322653s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.987396144 (637h2m35.584167177s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.997416565 (637h2m35.594187596s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005002081 (637h2m35.601773128s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005136838 (637h2m35.601907854s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005157484 (637h2m35.601928499s) [seg_pos={0 0}, upload=713386]: Satisfied cohort requirements after 13 pieces Apr 2 00:02:38.005386492 (637h2m35.602157525s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005396281 (637h2m35.60216731s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005531231 (637h2m35.602302263s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005570651 (637h2m35.602341684s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005692951 (637h2m35.602463983s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005700918 (637h2m35.602471942s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005722537 (637h2m35.60249356s) [seg_pos={0 0}, upload=713386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 2 00:02:38.005805596 (637h2m35.60257662s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:02:38.005812420 (637h2m35.602583444s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:03:07.848626111 (637h3m5.445397142s) [upload=713386]: Done uploading segments... Apr 2 00:03:07.848811089 (637h3m5.445582113s) [upload=713386]: Upload finished. err: metaclient: context deadline exceeded Duration: 491.89991ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................XX.XXX...X.............................] duration: 1.71471027s trace: https://jaeger.dev.storj.io/trace/7b66886c63182d02 FAIL (2026-04-02 00:32:14 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:08:32.201995061 (637h8m29.798766091s) [upload=713490]: Starting upload Apr 2 00:08:32.202058979 (637h8m29.798830011s) [upload=713490]: Done starting upload Apr 2 00:08:32.202104136 (637h8m29.798875153s) [upload=713490]: Uploading segments... Apr 2 00:08:32.202468215 (637h8m29.799239227s) [upload=713490]: Got next segment. Inline: false Apr 2 00:08:32.202545730 (637h8m29.799316756s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:08:32.202554537 (637h8m29.799325554s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:08:32.236884387 (637h8m29.83365542s) [seg_pos={0 0}, upload=713490]: Begin upload segment... Apr 2 00:08:32.236972300 (637h8m29.833743327s) [seg_pos={0 0}, upload=713490]: Done begin upload segment. Apr 2 00:08:32.237040287 (637h8m29.833811319s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237609577 (637h8m29.834380609s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237761417 (637h8m29.834532443s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.238323459 (637h8m29.835094484s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239091520 (637h8m29.835862549s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239476832 (637h8m29.836247858s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240185307 (637h8m29.836956336s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240585259 (637h8m29.837356286s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241120411 (637h8m29.83789144s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241483943 (637h8m29.838254967s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241758194 (637h8m29.838529219s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.242223914 (637h8m29.838994946s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243144784 (637h8m29.839915821s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243610121 (637h8m29.840381134s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244322341 (637h8m29.841093371s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244793070 (637h8m29.841564104s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245167641 (637h8m29.841938667s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245545293 (637h8m29.842316322s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.309369659 (637h8m29.9061407s) [upload=713490]: Next returned nil segment Apr 2 00:08:32.309386955 (637h8m29.906157989s) [upload=713490]: Waiting for error group managing segments... Apr 2 00:08:32.314998992 (637h8m29.911770022s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.317608657 (637h8m29.914379694s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.331442646 (637h8m29.92821367s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.334324786 (637h8m29.931095819s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.347843004 (637h8m29.944614014s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.349783033 (637h8m29.946554099s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.449314364 (637h8m30.04608539s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.466602858 (637h8m30.063373882s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.620540338 (637h8m30.217311348s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.627266502 (637h8m30.224037516s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.633186725 (637h8m30.229957751s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.836095281 (637h8m30.432866306s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.841319166 (637h8m30.438090195s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852957995 (637h8m30.449729018s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852982095 (637h8m30.449753118s) [seg_pos={0 0}, upload=713490]: Satisfied cohort requirements after 14 pieces Apr 2 00:08:32.853223544 (637h8m30.449994576s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853232550 (637h8m30.450003573s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853611136 (637h8m30.450382165s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MVWRaM417kW63ffaSvNzgtk1PxPQbbvDAxEzFSLVjDkhkLt7f): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853619602 (637h8m30.450390625s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853821693 (637h8m30.450592723s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GihGm5Lo1EDs1ZMqctuqxN3VVpvZHz7iBo4aSvhRhF2sBkAKx): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853828723 (637h8m30.450599745s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853975694 (637h8m30.450746723s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853981928 (637h8m30.450752951s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.854004392 (637h8m30.450775415s) [seg_pos={0 0}, upload=713490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 2 00:08:32.854137807 (637h8m30.450908831s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:08:32.854144625 (637h8m30.450915648s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:08:32.919468065 (637h8m30.516239079s) [upload=713490]: Done uploading segments... Apr 2 00:08:32.919489033 (637h8m30.516260054s) [upload=713490]: Upload finished. err: <nil> Duration: 1.714810947s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 10 out of 60 is failed (threshold=3) [...............XXXXXX.XXX........X..........................] duration: 250.574636ms trace: https://jaeger.dev.storj.io/trace/2d309a7a06fb1cdc FAIL (2026-04-02 00:32:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:12:21.530762169 (637h12m19.127533179s) [upload=713560]: Starting upload Apr 2 00:12:21.530842522 (637h12m19.127613558s) [upload=713560]: Done starting upload Apr 2 00:12:21.530951258 (637h12m19.127722284s) [upload=713560]: Uploading segments... Apr 2 00:12:21.531087313 (637h12m19.127858578s) [upload=713560]: Got next segment. Inline: true Apr 2 00:12:21.531098521 (637h12m19.127869553s) [upload=713560]: Waiting for error group managing segments... Apr 2 00:12:21.531173202 (637h12m19.127944216s) [upload=713560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:12:21.531178995 (637h12m19.127950018s) [upload=713560]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Apr 2 00:12:21.531183386 (637h12m19.12795441s) [upload=713560]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:13:21.491464871 (637h13m19.088235905s) [upload=713560]: Done uploading segments... Apr 2 00:13:21.491522533 (637h13m19.088293572s) [upload=713560]: Upload finished. err: metaclient: context deadline exceeded Duration:* 250.677424ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 487.660331ms trace: https://jaeger.dev.storj.io/trace/3c28cdfe17d1a942 FAIL (2026-04-02 00:31:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:02:37.851070386 (637h2m35.44784141s) [upload=713386]: Starting upload Apr 2 00:02:37.851166355 (637h2m35.447937379s) [upload=713386]: Done starting upload Apr 2 00:02:37.851253090 (637h2m35.44802412s) [upload=713386]: Uploading segments... Apr 2 00:02:37.851793181 (637h2m35.448564196s) [upload=713386]: Got next segment. Inline: false Apr 2 00:02:37.851882102 (637h2m35.448653125s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:02:37.851889461 (637h2m35.448660484s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:02:37.880067208 (637h2m35.476838238s) [seg_pos={0 0}, upload=713386]: Begin upload segment... Apr 2 00:02:37.880168602 (637h2m35.476939634s) [seg_pos={0 0}, upload=713386]: Done begin upload segment. Apr 2 00:02:37.880207987 (637h2m35.476979021s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880214330 (637h2m35.476985339s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880292505 (637h2m35.477063531s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881027652 (637h2m35.477798688s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881378064 (637h2m35.478149089s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882097992 (637h2m35.478869026s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882586634 (637h2m35.479357658s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.883643537 (637h2m35.48041457s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884145002 (637h2m35.480916034s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884766674 (637h2m35.481537743s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885263298 (637h2m35.482034334s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885763219 (637h2m35.482534252s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886298227 (637h2m35.483069261s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886892344 (637h2m35.483663375s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.887518235 (637h2m35.484289271s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.888216955 (637h2m35.484987987s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.976004543 (637h2m35.572775567s) [upload=713386]: Next returned nil segment Apr 2 00:02:37.976025238 (637h2m35.572796273s) [upload=713386]: Waiting for error group managing segments... Apr 2 00:02:37.982357614 (637h2m35.579128639s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.982449885 (637h2m35.5792209s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983162285 (637h2m35.579933315s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983404595 (637h2m35.580175625s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983964188 (637h2m35.580735217s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.984025323 (637h2m35.58079634s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.985346526 (637h2m35.582117555s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986081391 (637h2m35.582852422s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986551623 (637h2m35.583322653s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.987396144 (637h2m35.584167177s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.997416565 (637h2m35.594187596s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005002081 (637h2m35.601773128s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005136838 (637h2m35.601907854s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005157484 (637h2m35.601928499s) [seg_pos={0 0}, upload=713386]: Satisfied cohort requirements after 13 pieces Apr 2 00:02:38.005386492 (637h2m35.602157525s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005396281 (637h2m35.60216731s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005531231 (637h2m35.602302263s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005570651 (637h2m35.602341684s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005692951 (637h2m35.602463983s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005700918 (637h2m35.602471942s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005722537 (637h2m35.60249356s) [seg_pos={0 0}, upload=713386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 2 00:02:38.005805596 (637h2m35.60257662s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:02:38.005812420 (637h2m35.602583444s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:03:07.848626111 (637h3m5.445397142s) [upload=713386]: Done uploading segments... Apr 2 00:03:07.848811089 (637h3m5.445582113s) [upload=713386]: Upload finished. err: metaclient: context deadline exceeded Duration: 487.686315ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX............] duration: 619.901986ms trace: https://jaeger.dev.storj.io/trace/7766953e0b42057d FAIL (2026-04-02 00:31:29 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 619.952729ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................XX.XXX...X............................] duration: 1.255449248s trace: https://jaeger.dev.storj.io/trace/7b66886c63182d02 FAIL (2026-04-02 00:31:27 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:08:32.201995061 (637h8m29.798766091s) [upload=713490]: Starting upload Apr 2 00:08:32.202058979 (637h8m29.798830011s) [upload=713490]: Done starting upload Apr 2 00:08:32.202104136 (637h8m29.798875153s) [upload=713490]: Uploading segments... Apr 2 00:08:32.202468215 (637h8m29.799239227s) [upload=713490]: Got next segment. Inline: false Apr 2 00:08:32.202545730 (637h8m29.799316756s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:08:32.202554537 (637h8m29.799325554s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:08:32.236884387 (637h8m29.83365542s) [seg_pos={0 0}, upload=713490]: Begin upload segment... Apr 2 00:08:32.236972300 (637h8m29.833743327s) [seg_pos={0 0}, upload=713490]: Done begin upload segment. Apr 2 00:08:32.237040287 (637h8m29.833811319s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237609577 (637h8m29.834380609s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237761417 (637h8m29.834532443s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.238323459 (637h8m29.835094484s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239091520 (637h8m29.835862549s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239476832 (637h8m29.836247858s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240185307 (637h8m29.836956336s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240585259 (637h8m29.837356286s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241120411 (637h8m29.83789144s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241483943 (637h8m29.838254967s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241758194 (637h8m29.838529219s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.242223914 (637h8m29.838994946s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243144784 (637h8m29.839915821s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243610121 (637h8m29.840381134s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244322341 (637h8m29.841093371s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244793070 (637h8m29.841564104s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245167641 (637h8m29.841938667s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245545293 (637h8m29.842316322s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.309369659 (637h8m29.9061407s) [upload=713490]: Next returned nil segment Apr 2 00:08:32.309386955 (637h8m29.906157989s) [upload=713490]: Waiting for error group managing segments... Apr 2 00:08:32.314998992 (637h8m29.911770022s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.317608657 (637h8m29.914379694s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.331442646 (637h8m29.92821367s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.334324786 (637h8m29.931095819s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.347843004 (637h8m29.944614014s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.349783033 (637h8m29.946554099s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.449314364 (637h8m30.04608539s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.466602858 (637h8m30.063373882s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.620540338 (637h8m30.217311348s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.627266502 (637h8m30.224037516s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.633186725 (637h8m30.229957751s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.836095281 (637h8m30.432866306s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.841319166 (637h8m30.438090195s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852957995 (637h8m30.449729018s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852982095 (637h8m30.449753118s) [seg_pos={0 0}, upload=713490]: Satisfied cohort requirements after 14 pieces Apr 2 00:08:32.853223544 (637h8m30.449994576s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853232550 (637h8m30.450003573s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853611136 (637h8m30.450382165s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MVWRaM417kW63ffaSvNzgtk1PxPQbbvDAxEzFSLVjDkhkLt7f): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853619602 (637h8m30.450390625s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853821693 (637h8m30.450592723s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GihGm5Lo1EDs1ZMqctuqxN3VVpvZHz7iBo4aSvhRhF2sBkAKx): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853828723 (637h8m30.450599745s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853975694 (637h8m30.450746723s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853981928 (637h8m30.450752951s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.854004392 (637h8m30.450775415s) [seg_pos={0 0}, upload=713490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 2 00:08:32.854137807 (637h8m30.450908831s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:08:32.854144625 (637h8m30.450915648s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:08:32.919468065 (637h8m30.516239079s) [upload=713490]: Done uploading segments... Apr 2 00:08:32.919489033 (637h8m30.516260054s) [upload=713490]: Upload finished. err: <nil> Duration: 1.255594956s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 480.522199ms trace: https://jaeger.dev.storj.io/trace/3c28cdfe17d1a942 FAIL (2026-04-02 00:31:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:02:37.851070386 (637h2m35.44784141s) [upload=713386]: Starting upload Apr 2 00:02:37.851166355 (637h2m35.447937379s) [upload=713386]: Done starting upload Apr 2 00:02:37.851253090 (637h2m35.44802412s) [upload=713386]: Uploading segments... Apr 2 00:02:37.851793181 (637h2m35.448564196s) [upload=713386]: Got next segment. Inline: false Apr 2 00:02:37.851882102 (637h2m35.448653125s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:02:37.851889461 (637h2m35.448660484s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:02:37.880067208 (637h2m35.476838238s) [seg_pos={0 0}, upload=713386]: Begin upload segment... Apr 2 00:02:37.880168602 (637h2m35.476939634s) [seg_pos={0 0}, upload=713386]: Done begin upload segment. Apr 2 00:02:37.880207987 (637h2m35.476979021s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880214330 (637h2m35.476985339s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880292505 (637h2m35.477063531s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881027652 (637h2m35.477798688s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881378064 (637h2m35.478149089s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882097992 (637h2m35.478869026s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882586634 (637h2m35.479357658s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.883643537 (637h2m35.48041457s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884145002 (637h2m35.480916034s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884766674 (637h2m35.481537743s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885263298 (637h2m35.482034334s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885763219 (637h2m35.482534252s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886298227 (637h2m35.483069261s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886892344 (637h2m35.483663375s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.887518235 (637h2m35.484289271s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.888216955 (637h2m35.484987987s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.976004543 (637h2m35.572775567s) [upload=713386]: Next returned nil segment Apr 2 00:02:37.976025238 (637h2m35.572796273s) [upload=713386]: Waiting for error group managing segments... Apr 2 00:02:37.982357614 (637h2m35.579128639s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.982449885 (637h2m35.5792209s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983162285 (637h2m35.579933315s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983404595 (637h2m35.580175625s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983964188 (637h2m35.580735217s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.984025323 (637h2m35.58079634s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.985346526 (637h2m35.582117555s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986081391 (637h2m35.582852422s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986551623 (637h2m35.583322653s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.987396144 (637h2m35.584167177s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.997416565 (637h2m35.594187596s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005002081 (637h2m35.601773128s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005136838 (637h2m35.601907854s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005157484 (637h2m35.601928499s) [seg_pos={0 0}, upload=713386]: Satisfied cohort requirements after 13 pieces Apr 2 00:02:38.005386492 (637h2m35.602157525s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005396281 (637h2m35.60216731s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005531231 (637h2m35.602302263s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005570651 (637h2m35.602341684s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005692951 (637h2m35.602463983s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005700918 (637h2m35.602471942s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005722537 (637h2m35.60249356s) [seg_pos={0 0}, upload=713386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 2 00:02:38.005805596 (637h2m35.60257662s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:02:38.005812420 (637h2m35.602583444s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:03:07.848626111 (637h3m5.445397142s) [upload=713386]: Done uploading segments... Apr 2 00:03:07.848811089 (637h3m5.445582113s) [upload=713386]: Upload finished. err: metaclient: context deadline exceeded Duration: 480.638803ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 10 out of 60 is failed (threshold=3) [................XXXXXX.XXX........X.........................] duration: 216.072721ms trace: https://jaeger.dev.storj.io/trace/2d309a7a06fb1cdc FAIL (2026-04-02 00:31:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:12:21.530762169 (637h12m19.127533179s) [upload=713560]: Starting upload Apr 2 00:12:21.530842522 (637h12m19.127613558s) [upload=713560]: Done starting upload Apr 2 00:12:21.530951258 (637h12m19.127722284s) [upload=713560]: Uploading segments... Apr 2 00:12:21.531087313 (637h12m19.127858578s) [upload=713560]: Got next segment. Inline: true Apr 2 00:12:21.531098521 (637h12m19.127869553s) [upload=713560]: Waiting for error group managing segments... Apr 2 00:12:21.531173202 (637h12m19.127944216s) [upload=713560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:12:21.531178995 (637h12m19.127950018s) [upload=713560]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Apr 2 00:12:21.531183386 (637h12m19.12795441s) [upload=713560]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:13:21.491464871 (637h13m19.088235905s) [upload=713560]: Done uploading segments... Apr 2 00:13:21.491522533 (637h13m19.088293572s) [upload=713560]: Upload finished. err: metaclient: context deadline exceeded Duration:* 216.097925ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......................XX.XXX...X...........................] duration: 1.128820369s trace: https://jaeger.dev.storj.io/trace/7b66886c63182d02 FAIL (2026-04-02 00:30:55 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:08:32.201995061 (637h8m29.798766091s) [upload=713490]: Starting upload Apr 2 00:08:32.202058979 (637h8m29.798830011s) [upload=713490]: Done starting upload Apr 2 00:08:32.202104136 (637h8m29.798875153s) [upload=713490]: Uploading segments... Apr 2 00:08:32.202468215 (637h8m29.799239227s) [upload=713490]: Got next segment. Inline: false Apr 2 00:08:32.202545730 (637h8m29.799316756s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:08:32.202554537 (637h8m29.799325554s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:08:32.236884387 (637h8m29.83365542s) [seg_pos={0 0}, upload=713490]: Begin upload segment... Apr 2 00:08:32.236972300 (637h8m29.833743327s) [seg_pos={0 0}, upload=713490]: Done begin upload segment. Apr 2 00:08:32.237040287 (637h8m29.833811319s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237609577 (637h8m29.834380609s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.237761417 (637h8m29.834532443s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.238323459 (637h8m29.835094484s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239091520 (637h8m29.835862549s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.239476832 (637h8m29.836247858s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240185307 (637h8m29.836956336s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.240585259 (637h8m29.837356286s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241120411 (637h8m29.83789144s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241483943 (637h8m29.838254967s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.241758194 (637h8m29.838529219s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.242223914 (637h8m29.838994946s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243144784 (637h8m29.839915821s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.243610121 (637h8m29.840381134s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244322341 (637h8m29.841093371s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.244793070 (637h8m29.841564104s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245167641 (637h8m29.841938667s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.245545293 (637h8m29.842316322s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Uploading piece... Apr 2 00:08:32.309369659 (637h8m29.9061407s) [upload=713490]: Next returned nil segment Apr 2 00:08:32.309386955 (637h8m29.906157989s) [upload=713490]: Waiting for error group managing segments... Apr 2 00:08:32.314998992 (637h8m29.911770022s) [noise=true , address=145.79.227.3:4182 , piece_id=ITEWESHQM43QIQ5IP4AO2WOEY43FDR5SQIWQZUKXHJQ7HTWT4UHA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.317608657 (637h8m29.914379694s) [noise=true , address=145.79.227.23:4569 , piece_id=SB57T3OSH4CD2TUHOPSVLRF24UZCNRVMOQ3EAXDENSU3QHVS6UXQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.331442646 (637h8m29.92821367s) [noise=true , address=45.91.116.5:4321 , piece_id=J6Q35QBFEOMOZU4ORS4CMQULAT45HMT7G52OBQXZ472MPNFOXCRQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.334324786 (637h8m29.931095819s) [noise=true , address=45.91.116.14:4194 , piece_id=YXZDFCNCU2WLKFUBBTOIQORFAH5AXFZAWLSU6PLSG3XDJ5XS5QYA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.347843004 (637h8m29.944614014s) [noise=true , address=45.91.116.4:4948 , piece_id=C2PGGXQKXSQTZPP6Y5M4363IXLWPDUISKEIR5ZA37VGYUYT6RZMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.349783033 (637h8m29.946554099s) [noise=true , address=77.90.31.17:4797 , piece_id=BHRYWQCQTJMDXCYDNGH2BJKU5C7IWVVQZ7NESQPL35DHFSQNHZEA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.449314364 (637h8m30.04608539s) [noise=true , address=5.231.73.4:4015 , piece_id=VVAA4OTCMSB6Y2CQFQY2PIL3XDN3N7FIHJPLDHPEZS5GWIWHJN3A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.466602858 (637h8m30.063373882s) [noise=true , address=145.79.227.31:4811 , piece_id=BM2JFQOPTQ7K43MDM2KLZMV6D46P4CGRGC4JNPGAA74Y53VWI5VQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.620540338 (637h8m30.217311348s) [noise=true , address=167.17.72.70:4747 , piece_id=H74VCXNXDZ6CZM3AXUXP4GLWX7T37DE7HR57QHAL43DAL6OHSWMQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.627266502 (637h8m30.224037516s) [noise=true , address=167.17.72.68:4659 , piece_id=NGP4NOS3W7OLEEPIZYZHW5X6JJZF5ZICWHDZIUY6HUZIXPGASFRA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.633186725 (637h8m30.229957751s) [noise=true , address=167.17.72.71:4789 , piece_id=J24TUILBCQFKGF4SPZSSHHTLH4GPMJQMP7UUFJZPONCY3UT7YXNA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.836095281 (637h8m30.432866306s) [noise=true , address=193.149.164.67:4008 , piece_id=2XXJEDWM5OP46UW472AAGNLD663JMW72OGR5PQHF64LR5IB3C3XA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.841319166 (637h8m30.438090195s) [noise=true , address=193.149.164.66:4981 , piece_id=Q67DSEBUQMRGFOKXFT3UHWSWX3NNLQ3YJHF67NMRGNMRPEIINSOA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852957995 (637h8m30.449729018s) [noise=true , address=193.149.164.71:4313 , piece_id=Z5T53CGA5FQFKIB2I2HJR3X6MGWHSIHX7KALQJFUDNFPHKQJQS2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: <nil> Apr 2 00:08:32.852982095 (637h8m30.449753118s) [seg_pos={0 0}, upload=713490]: Satisfied cohort requirements after 14 pieces Apr 2 00:08:32.853223544 (637h8m30.449994576s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853232550 (637h8m30.450003573s) [noise=true , address=77.90.31.15:4709 , piece_id=QBGHXPVTVC2HHP2AMVZ2ZQ27M3IA5ABSL6CIYIIAPNK76SQJMPXA, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853611136 (637h8m30.450382165s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MVWRaM417kW63ffaSvNzgtk1PxPQbbvDAxEzFSLVjDkhkLt7f): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853619602 (637h8m30.450390625s) [noise=true , address=64.49.13.139:4477 , piece_id=STCGSBBPHULYZWVXRXNKY55UDIO3LQGQGF53S6IHMRMRS2C7TMYQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853821693 (637h8m30.450592723s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GihGm5Lo1EDs1ZMqctuqxN3VVpvZHz7iBo4aSvhRhF2sBkAKx): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853828723 (637h8m30.450599745s) [noise=true , address=193.149.164.68:4036 , piece_id=K2XOHFEVMQXDQPFSC32JALQ3XKPM4CY3LVB2MHU5VAKMW2LSSY2A, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.853975694 (637h8m30.450746723s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Apr 2 00:08:32.853981928 (637h8m30.450752951s) [noise=true , address=145.79.227.33:4899 , piece_id=FA3OVYHIIG5SB42TMHUN2AV5VAL3GQBNHJKGK55OD5UATRNJ6IJQ, seg_pos={0 0}, upload=713490]: Long tail cancellation due to optimal threshold Apr 2 00:08:32.854004392 (637h8m30.450775415s) [seg_pos={0 0}, upload=713490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Apr 2 00:08:32.854137807 (637h8m30.450908831s) [upload=713490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:08:32.854144625 (637h8m30.450915648s) [upload=713490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:08:32.919468065 (637h8m30.516239079s) [upload=713490]: Done uploading segments... Apr 2 00:08:32.919489033 (637h8m30.516260054s) [upload=713490]: Upload finished. err: <nil> Duration: 1.128963067s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 543.083295ms trace: https://jaeger.dev.storj.io/trace/3c28cdfe17d1a942 FAIL (2026-04-02 00:30:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:02:37.851070386 (637h2m35.44784141s) [upload=713386]: Starting upload Apr 2 00:02:37.851166355 (637h2m35.447937379s) [upload=713386]: Done starting upload Apr 2 00:02:37.851253090 (637h2m35.44802412s) [upload=713386]: Uploading segments... Apr 2 00:02:37.851793181 (637h2m35.448564196s) [upload=713386]: Got next segment. Inline: false Apr 2 00:02:37.851882102 (637h2m35.448653125s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:02:37.851889461 (637h2m35.448660484s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 2 00:02:37.880067208 (637h2m35.476838238s) [seg_pos={0 0}, upload=713386]: Begin upload segment... Apr 2 00:02:37.880168602 (637h2m35.476939634s) [seg_pos={0 0}, upload=713386]: Done begin upload segment. Apr 2 00:02:37.880207987 (637h2m35.476979021s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880214330 (637h2m35.476985339s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.880292505 (637h2m35.477063531s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881027652 (637h2m35.477798688s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.881378064 (637h2m35.478149089s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882097992 (637h2m35.478869026s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.882586634 (637h2m35.479357658s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.883643537 (637h2m35.48041457s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884145002 (637h2m35.480916034s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.884766674 (637h2m35.481537743s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885263298 (637h2m35.482034334s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.885763219 (637h2m35.482534252s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886298227 (637h2m35.483069261s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.886892344 (637h2m35.483663375s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.887518235 (637h2m35.484289271s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.888216955 (637h2m35.484987987s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Uploading piece... Apr 2 00:02:37.976004543 (637h2m35.572775567s) [upload=713386]: Next returned nil segment Apr 2 00:02:37.976025238 (637h2m35.572796273s) [upload=713386]: Waiting for error group managing segments... Apr 2 00:02:37.982357614 (637h2m35.579128639s) [noise=true , address=145.79.227.39:4015 , piece_id=MZXYJ3JEWTQOXK2Q6D4D7UGZKKFLEUG377KR7HJURZPV6MLVPFKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.982449885 (637h2m35.5792209s) [noise=true , address=77.90.31.10:4522 , piece_id=FYFASDIM3L7XC6GS3Y4TLSDHWO3CMDUTLX44KTIUUNSZAGMNCOYQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983162285 (637h2m35.579933315s) [noise=true , address=145.79.227.10:4407 , piece_id=HX472PUEOULWYXXFIECA3WS25Y5I63N4F7IZJNGDJESTIJFEBKQA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983404595 (637h2m35.580175625s) [noise=true , address=145.79.227.6:4257 , piece_id=U35H5GJDO6QU2VFB6YDVAEV27JLVRXEFA6G3BABMKPOWWQGITLNA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.983964188 (637h2m35.580735217s) [noise=true , address=145.79.227.3:4153 , piece_id=B3NOUGBGQXXTZ4ZCQCVPIFGX6U4LJZUUR54ZF25Q7EQ6HDVBPZ4A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.984025323 (637h2m35.58079634s) [noise=true , address=145.79.227.25:4619 , piece_id=4VJVYMYLHCH2V3YH5CFZIWRRWLFKWI6IMGXIJXJ77FTMM5TEDDUA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.985346526 (637h2m35.582117555s) [noise=true , address=145.79.227.36:4522 , piece_id=UJEZ4AXAEGGQ53QMJSN7P3TFSIC7VWTQ2PUHDNPQY4VRX5HXETMQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986081391 (637h2m35.582852422s) [noise=true , address=145.79.227.24:4584 , piece_id=H6U7BDK2A56LYANKEKPOQSZ4HMOMMZBCABFPT3FNGEF3BF5IIOZA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.986551623 (637h2m35.583322653s) [noise=true , address=145.79.227.51:4414 , piece_id=FABNUCKRRFFPMBNJWJ4LGY3VLUJCCT6RAQQ5RR4HE76HLE2BXNKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.987396144 (637h2m35.584167177s) [noise=true , address=145.79.227.29:4944 , piece_id=V2CCGUNSNLYEZ6QEBQQXXBZYJRGAFL4YLFGO2D5ZCCPJHDBE72FQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:37.997416565 (637h2m35.594187596s) [noise=true , address=45.91.116.10:4081 , piece_id=LF4ADKKCRYQT5C3GE4QQN3QO5BRSVHGZ4ZKSPGICT7DHX7SV3JEA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005002081 (637h2m35.601773128s) [noise=true , address=77.90.31.13:4629 , piece_id=IVMXOROU2BL34EXEZLVHU7CCVRUUDJS42J2FED6O2IS567IFVA7A, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005136838 (637h2m35.601907854s) [noise=true , address=145.79.227.45:4234 , piece_id=YQDA6QFB3JWTMAIPUBB3HE7TE55XAHHEYOSLMI7JORH56FOMOQRA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: <nil> Apr 2 00:02:38.005157484 (637h2m35.601928499s) [seg_pos={0 0}, upload=713386]: Satisfied cohort requirements after 13 pieces Apr 2 00:02:38.005386492 (637h2m35.602157525s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005396281 (637h2m35.60216731s) [noise=true , address=145.79.227.8:4329 , piece_id=MA32EIMBQBG3RA3G54KWPIZ746TG5OX3EXGGUZ5NUTFJPKLXTUUQ, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005531231 (637h2m35.602302263s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005570651 (637h2m35.602341684s) [noise=true , address=145.79.227.37:4567 , piece_id=SVG7DQE5M7V7JMLOSJKH25SPTHBVRA4QBFOV7ZMV57CTIUKHUSKA, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005692951 (637h2m35.602463983s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Apr 2 00:02:38.005700918 (637h2m35.602471942s) [noise=true , address=77.90.31.12:4590 , piece_id=SOX2DI2RDVX2YKEBQM47L45PCEWN6ZWEDWFJB36ROSUNUKXKBT5Q, seg_pos={0 0}, upload=713386]: Long tail cancellation due to optimal threshold Apr 2 00:02:38.005722537 (637h2m35.60249356s) [seg_pos={0 0}, upload=713386]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Apr 2 00:02:38.005805596 (637h2m35.60257662s) [upload=713386]: Flush batch item: pb.BatchRequestItem_SegmentCommit Apr 2 00:02:38.005812420 (637h2m35.602583444s) [upload=713386]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:03:07.848626111 (637h3m5.445397142s) [upload=713386]: Done uploading segments... Apr 2 00:03:07.848811089 (637h3m5.445582113s) [upload=713386]: Upload finished. err: metaclient: context deadline exceeded Duration: 543.117748ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [.............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...........] duration: 728.221601ms trace: https://jaeger.dev.storj.io/trace/7766953e0b42057d FAIL (2026-04-02 00:30:29 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 728.252724ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 10 out of 60 is failed (threshold=3) [.................XXXXXX.XXX........X........................] duration: 226.085301ms trace: https://jaeger.dev.storj.io/trace/2d309a7a06fb1cdc FAIL (2026-04-02 00:30:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Apr 2 00:12:21.530762169 (637h12m19.127533179s) [upload=713560]: Starting upload Apr 2 00:12:21.530842522 (637h12m19.127613558s) [upload=713560]: Done starting upload Apr 2 00:12:21.530951258 (637h12m19.127722284s) [upload=713560]: Uploading segments... Apr 2 00:12:21.531087313 (637h12m19.127858578s) [upload=713560]: Got next segment. Inline: true Apr 2 00:12:21.531098521 (637h12m19.127869553s) [upload=713560]: Waiting for error group managing segments... Apr 2 00:12:21.531173202 (637h12m19.127944216s) [upload=713560]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 2 00:12:21.531178995 (637h12m19.127950018s) [upload=713560]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Apr 2 00:12:21.531183386 (637h12m19.12795441s) [upload=713560]: Flush batch item: pb.BatchRequestItem_ObjectCommit Apr 2 00:13:21.491464871 (637h13m19.088235905s) [upload=713560]: Done uploading segments... Apr 2 00:13:21.491522533 (637h13m19.088293572s) [upload=713560]: Upload finished. err: metaclient: context deadline exceeded Duration:* 226.12173ms core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)